Title: On the creation of narrow AI: hierarchy and nonlocality of neural network skills
Venue: NeurIPS 2025
Accepted: yes
Keywords: safety

mechanistic interpretability

theory

curriculum learning

pruning

connectionism

unlearning
Abstract: We study the problem of creating strong, yet narrow, AI systems. While recent AI progress has been driven by the training of large general-purpose foundation models, the creation of smaller models specialized for narrow domains could be valuable for both efficiency and safety. In this work, we explore two challenges involved in creating such systems, having to do with basic properties of how neural networks learn and structure their representations. The first challenge regards when it is possible to train narrow models from scratch. Through experiments on a synthetic task, we find that it is sometimes necessary to train networks on a wide distribution of data to learn certain narrow skills within that distribution. This effect arises when skills depend on each other hierarchically, and training on a broad distribution introduces a curriculum which substantially accelerates learning. The second challenge regards how to transfer particular skills from large general models into small specialized models. We find that model skills are often not perfectly localized to a particular set of prunable components. However, we find that methods based on pruning can still outperform distillation. We investigate the use of a regularization objective to align desired skills with prunable components while unlearning unnecessary skills.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission investigates the creation of narrow, specialized models. The authors argue that hierarchical tasks in some cases require broad pre-training of larger models. Experiments on synthetic data show that such cases can exist. Further, the authors evaluate whether specialized knowledge can be localized in the weights of models, and could be isolated by pruning the other weights. They find that specialized knowledge is entangled in weights, but can be dis-entangled by fine-tuning on the target data with a group-lasso regularization.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n+ The motivation of narrow models is convincing, especially via efficiency. \n\n+ The paper presents an interesting case study on learning dynamics of compositional tasks. While the results are intuitive (atomic tasks learned first, composite tasks learned after), I wonder if the experiments conflates compositionality with 'hardness' of the task and can support the conclusion of hierarchical learning. Presumably, compositional tasks are significantly harder and so will be picked up later. To get clean results, the two need to be disentangled. I'm not sure if CMSP allows for that.\n\n+ Albeit not entirely surprising, the combination of fine-tuning with l1 regularization and subsequent pruning is a reasonable idea, and the experiments support the combination. I'm not sure if similar ideas have been tried before since it seems such a natural combination, but its interesting that localization can be strenghtend post-hoc even on pre-trained models.\n\n+ The submission is generally clear and well written. Experiments are well motivated. I especially appreciate that the authors don't embellish their results and discuss limitations.\n\n### Weaknesses\n\n- The motivation via mechanistic interpretability or verifiability is not convincing to me. Differences between full and narrow models are probably in scale, not in general composition. Particularly in cases like coding task, where models are qualitatively similar. Hence, methods that work on narrow models can also be expected to work on general models.\n\n- The authors make a strong assumption that (some) skills are hierarchical. This is not my field, but I keep wondering. Hierarchical skills exist - but is there actual evidence for it? Is that relevant in practice, or is this a notion carried over from what and how humans learn? Even if it exists, its unclear to what extend the CMSP experiments represent real-world compositionally and how far findings might carry over. The authors touch upon that limitation in line 159. \n\n- While I like Figure 5, I feel it only shows part of a picture. Specifically if random pruning achieves similar performance, it appears that the overall computational load might be lowest to just randomly prune to the desired sparsity and follow up with recovery steps. The group-lasso steps appear to keep the loss low for a low recovery step regime, but afterwards at best match the baseline. \n\n- I keep wondering how the two parts of the paper are connected. The first considers hierarchical skills and how easy / hard they are to learn. The second studies extracting skills from a larger model into a smaller one. The two appear disjunct, albeit they may both be part of a larger pipeline. If the argument the authors wish to make is that one should pre-train broadly to learn complex composite skills faster, and then prune aggressively, I would encourage them to tie the two together more tightly in an overarching narrative. In the second half, the evaluation uses a Llama on code. Maybe running experiments on hierarchical skills in that domain might help to form a coherent narrative?\n\nQUESTIONS:\nSee strengths and weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the problem of building narrow AI systems, models specialized for a single task or domain, by investigating two core challenges: 1) when can a narrow model be trained from scratch 2) how do you transfer specific skills from a pretrained general model.\n\nFor challenge 1, the authors introduce a synthetic benchmark task, compositional multitask sparse parity (CMSP), designed to exhibit hierarchical dependencies among subtasks. Using CMSP, they show that training on broader data distributions can enable more efficient learning of composite tasks.\n\nFor challenge 2 the paper compares methods for narrowing models with a focus on pruning, distillation and group lasso regularisation - a regularisation method introduced by the authors to encourage the model to learn tasks in a way that better supports pruning. The authors report that pruning + recovery training (continued training on the desired task) often outperforms distillation and training from scratch at preserving desired skills while reducing model capacity. Experiments are conducted on CMSP, MNIST and Python documentation using LLMs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper focuses on narrow AI systems, something that is less common after the recent successes of general-purpose models. This is still an important area of research as we have evidence from biological agents of the benefit of the combination of smaller, focused models and so definitely deserves further exploration.\n- The CMSP benchmark nicely introduces a hierarchical dependency between tasks and can act as a useful tool for future investigations of tasks with this very hierarchical structure.\n- The paper focuses on an interesting problem that is often pushed aside in current works, the overlapping neuron representations of different skills in larger models, which can make pruning challenging and reduces explainability.\n- The paper is laid out well, building both the narrative of their method and the use of their experiments to help the reader understand both the challenges of this problem and the proposed solutions.\n\n**Weaknesses**\n- The paper claims to explore when it is possible to train narrow models from scratch and concludes that \"Our work shows that for some types of tasks, it may be necessary to train on a broad data distribution in order to learn.\" The authors only investigated one type of task structure - highly hierarchical - and found you needed a broad dataset to efficiently learn. This challenge seems poorly investigated and doesn't result in a clear answer.\n- The paper has a heavy focus on results from the synthetic task CMSP. Additionally, they do not look at the effect of their proposed methods on non-hierarchical baselines. The mention two other synthetic tasks that they drew inspiration from which they could have added to their evaluation. While you could argue that the papers that originally introduced these benchmarks already explored the effects of learning these synthetic tasks, it is not clear in the paper that the aim of these investigations was to investigate the learning of narrow tasks which is what this work wishes to explore. The two non-synthetic experiments, MNIST and LLMs on Python documentation, are only briefly mentioned and only offer that pruning works better than distillation without much insight into these findings. Additionally, the group lasso regularisation is not used on the LLM experiment.\n- The paper has a few areas where a speculative reason is provided to explain some phenomena and is then later used as the grounding for a future speculative explanation. For example \"One explanation of these results is that networks compute the parity of the atomic subtask bits in the first hidden layer, and then to learn the composite subtask they compute the parity of these values in the second layer.\" However, this mechanism is never empirically validated. No activation probing, representational analysis, or layer-specific ablation is performed to support this claim. Instead, a comparison between shallow and deeper networks is offered, but this is insufficient to rule out the impact of increased expressivity due to added depth. While a width-matched baseline is included in the appendix, it does not eliminate the core confounding point that deeper networks may simply represent more complex functions by design. This statement is later used to add legitimacy to the broader claim \"This result may also provide some explanation for advantage of depth in deep learning – not only is depth necessary for networks to efficiently approximate certain functions [33], we find here that depth is helpful to efficiently learn tasks with hierarchical structure.\" The use of one unproven explanation as evidence for another raises concerns about the paper’s reasoning rigor in the discussion section.\n- The paper could greatly benefit from another look at the figures and their captions. For example figure 3 has visualizations of 2-hidden-layer MLPs to provide evidence of the use of group lasso regularisation however an explanation for the lines is never given (what do the colours represent? Why is the line thickness different? Is this weight magnitude? Activation size?). I find figure 4 left difficult to parse as the authors have a three dimensional result which they are showing in a 2D graph so gleaning any real patterns requires a lot of thought on the readers side. Figure 5 right has shadow lines but no explanation is given for what this is. It cannot be an STD or error estimate because it is offset from the main lines in some cases and I can't even guess what these shadow lines mean.\n- The limitations section of this paper is poor and lacks an honest self-critique. The authors point out they only look at one, simple pruning strategy and lack of LLM hyperparameter tuning as the only limitations in their work. They then have another paragraph that raises philosophical challenges related to narrow AI and postulate that this entire direction of work could be irrelevant if neural networks learn in a way that would negate this subfield of research. This is not a useful limitations section and ignores some real limitations of the presented work.\n\nQUESTIONS:\n- Have you done any exploration into tasks that have a hierarchical structure in the presentation of the data but not necessarily the task? For example the use of abstractions in image based tasks, where there is an underlying structure that can make the task easier but is not directly tied to the result.\n- By removing the ability to recover performance on a different task, is it possible to degrade performance on your current task if they are closely linked? This seems to me to be a bit of a random proxy to be optimizing for that could possibly lead to degradation of the whole system if done in some settings.\n- As the paper stands, I would raise my score if the figures are better labelled and explained, the language and discussion was made a bit more rigorous and if the limitations section was a more critical reflection of the work actually done in this paper.\n\nLIMITATIONS:\nThe authors did not efficiently evaluate the limitations of their work. They show the \"simple pruning strategy\" can outperform distillation so claiming this as a limitation is a bit hollow. Similarly, the LLM is used as a baseline so while the results may change if the LLM was well tuned the lessons learnt from the LLM experiment should still hold true. The rest of the limitations is just philosophical musings that are more a statement about the subfield of hierarchical learning and not about the work. Instead the authors could have mentioned their lack of task diversity in their synthetic study, lack of evaluation of group lasso regularisation in the LLM experiment and the fact that much of their analysis is done only on synthetic tasks.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main question that this paper aims to answer is:\n\nwhat if we try to create a specialized (\"narrow\") model by extracting specific skills from a more general-purpose larger model? Which are the main challenges in doing so?\n\nThe authors focus on two specific challenges: \n1) some narrow tasks will require broad training curricula because of hierarchical dependencies between tasks. They explore this issue using a synthetic task called CMSP (compositional multitask sparse parity -- it is a variation of a task that another proposed earlier). \n2) nonlocality of representations: the problem here is it is hard to isolate specific skills through direct pruning because of the distributed way in which neural networks learn representations.\n\nThe paper presents some structured pruning methods (that are not new themselves -- e.g., group-lasso regularization) to extract and transfer narrow skill subsets into smaller networks.\n\nSTRENGTHS AND WEAKNESSES:\nMain strengths\n* The topic (at a high level) is significant because extracting narrow, specialized AI models (through pruning, distillation, etc) efficiently from general-purpose models is a practically important area for scaling up AI systems.\n* The paper’s experimental setup is clearly described. Figures and visualizations are informative and well done.\n* The writing is also clear and good.\n\nMain Weaknesses\n* In my opinion, the paper does not have a very clear result or \"message\". It reads more as an \"exploration report\",  showing some challenges in extracting narrow models from more general-purpose models -- but it is not clear at the end of the day whether these challenges would still exist if the \"extraction process\" was different. The conclusions of the paper are specific to the specific synthetic hierarchical task that is considered and the specific pruning methods that are considered.\n* There is a lack of theoretical or clear analytical results explaining why and how hierarchical curriculum learning and non-locality arise from neural network training. The results rely on empirical observations.\n* The main insights (hierarchical curriculum learning, nonlocality, the use of structured pruning methods) do not come across as highly original. These are well understood concepts in the broader neural network literature.\n\nQUESTIONS:\n* I suggest that the authors read the following paper:\nhttps://arxiv.org/pdf/2406.08658  (Pruning is Optimal for Learning Sparse Features in High-Dimensions)\nIt is a more theoretical paper but I think it is very relevant.\n\n* One of the paper's main claims is that the  distributed nature of representations is a significant challenge for pruning. However the pruning can be done during training or fine-tuning for that more narrow/specific task. Would such an approach challenge this main claim?\n\n* The paper provides some limited results for experiments involving LLMs (Llama), evaluating on Python token prediction. I admit that I found that part of the paper rather speculative and not very convincing. For example, the authors state about Fig-6 \"we tentatively find that pruning substantially outperforms training from scratch and distilling a model from scratch on the data-parameter frontier\" -- or later in the same section: \n\"in our case it does not seem like the relevant features for our task are localized into a set of “Python” vs.“non-Python” neurons, at least that attribution pruning identifies.\"\nIt is important to have such experiments because they bring the paper closer to the \"real world\" -- but it is also necessary for these experiments to be done more deeply/extensively and the analysis of the results to be more conclusive.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the challenges in creating _narrow_ AI: \"tool\" AI systems designed to have specific skills as opposed to general-purpose models with broad skills. This is motivated from the viewpoints of information efficiency as well as preservation of human agency. The study is conducted from two broad angles: (1) the efficacy of training a narrowly-skilled model from scratch, and (2) how best one could transfer specific skills (while discarding the rest) from a generalist model into a specialist model.\n\nRegarding (1), an empirical study using the compositional multitask sparse parity (CMSP) task is conducted. The task is designed such that certain difficult subtasks are \"composites\" of easier, \"atomic\" subtasks, inducing the possibility for \"curriculum learning\". It is observed that neural networks indeed learn the compositional task much faster when also trained on the atomic tasks, while training from scratch only on the compositional \"narrow\" subset is sample-inefficient (i.e., requires more samples in total to reach the same performance) -- this indicates that those easier skills cannot be safely skipped when training from scratch. They also find that depth helps learning to solve CMSP, leading to the hypothesis that many deep networks indeed learn hierarchical tasks in the same staggered fashion. \n\nRegarding (2), pruning and distillation are explored as candidate methods for selectively transferring skills from a broader model into a smaller model. This is analysed using the CMSP task above, MNIST digit classification, as well as LLM next-token prediction. They find that the circuitries for different tasks are entangled: this makes pruning for specific skills a non-trivial task. To address this problem, they propose a group lasso regularization that aims to \"clean up\" entire groups of neurons that are irrelevant to the narrow task at hand. The observations suggest that pruning the model whilst applying the proposed regularization, is more efficient in comparison to distillation.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. The questions explored are insightful, interesting and well-motivated. With general-purpose MLLMs taking over the world, this kind of work that explores specialist models instead, is well-timed.\n1. The proposed compositional multitask sparse parity task and the group lasso regularization for pruning can, in themselves, be valuable for future work. Notably, pruning with the sparsity regularization seems to be an effective strategy for narrow skill transfer. \n1. The paper is fairly well-organized given the density of information covered. Sections follow each other logically.\n\n### Weaknesses\n\n1. The curriculum learning claims are based entirely on the toy case study of compositional multitask sparse parity.  While these experiments themselves are quite thorough, they do not guarantee transfer to real-world cases: the authors note this in §3.2. Limitations noted in §7 also apply. \n1. The authors conduct LLM experiments in §5, but in contrast, the vision counterpart of these experiments in §4 is confined to MNIST. Given that vision is just as important as language in modern AI systems, I would be curious to see whether the results transfer to more realistic vision datasets -- absence of these results does impact the paper's significance. \n\nOverall, I believe the strengths sufficiently outweigh the weaknesses: enough for me to recommend borderline acceptance at this stage. I would have considered a higher score if more realistic-dataset experiments had been presented, but I would be happy to reconsider my score during the discussion phase and after looking at insights from other reviewers.\n\nQUESTIONS:\nDo the authors have any ideas/plans for extending the experimental setup in §3 and §4 to more realistic datasets?\n\nLIMITATIONS:\nThe authors mention limitations in §7, but should mention that CMSP is a toy task (which they do mention in lines 159-162 but should also briefly include in §7 for completeness).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}