Kati8/ICLR_2023_2025 huggingface dataset:
authors: Siddhartha Datta
authors: Nigel Shadbolt
reviews: 
Comments: The research delved into the challenges large models face when trying to generalize in zero-shot and few-shot situations during continual learning. Unlike previous studies that mainly focused on performance memory within domain-specific tasks, this study introduces a technique to maintain performance for both in-domain and out-of-domain tasks.
Strengths: SNP surpasses other benchmarks in multiple tasks, including zero-shot and few-shot learning.
Weaknesses: (1) The authors' approach to the continual learning problem is ambiguous. I'm uncertain about the specific issue they're addressing, and I'm concerned that the authors might also lack clarity on the subject.

(2) The objective function defines base parameters using all the tasks in the set. Yet, this conflicts with the fact that tasks are presented to the learner sequentially. How can a base learner be defined using all the data when it only becomes accessible at the end of its operation? This inconsistency suggests a fundamental flaw in the framework.

(3) It seems that task groupings are determined using data from all tasks in Sec. 3.1. This suggests the experimental design relies on complete task data, making it questionable. A sound experimental design should use randomized trials or employ specific methodologies to assess and counteract biases from non-random setups. However, it appears the authors recognize biases in the data and use them to shape their experimental approach. This methodology is problematic and lacks validity.

(4) Regarding the presentation, the placement of tables separate from their explanatory text makes them challenging to follow. A suggestion for improved clarity would be to bold the top-performing metrics in each table corresponding to specific tasks.
Recommendation: 3: reject, not good enough
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 1 poor
reviews: 
Comments: The paper tackles the problem of learning with limited labels in the context of continual learning. It develops the approach based on subnetworks whose parameters are represented in the same space as base learner parameters, resulting in learning composable and editable network representations. I think overall this is an interesting idea, akin to hypernetworks. However, the paper does not demonstrate that the derived representations indeed possess the claimed properties. Moreover, the presentation of the paper needs significant improvement, something that would require a major rewrite of the entire manuscript. Although I see value in the idea and empirical results, the revision that is necessary to make the paper fit quality criteria of a conference such as ICLR, in my opinion, is well beyond what is typically achievable during a conference paper revision. Therefore I recommend reject.
Strengths: - interesting and exciting topic
- interesting results comparing CLIP and MAML
- extensive empirical results
Weaknesses: - I believe there are some typos in equations, or otherwise the descriptions are not clear
- Writing clarity, structure and flow need to be improved
- the setup seems compromised as the accuracy in unseen datasets is used to guide model training
- the paper is written as a technical report describing what was done. for a scientific paper, I would expect a lot more high-level 
- motivatioon of design and theory supporting the algorithm development.
- The paper claims that the subnetwork representations can be added, removed, edited, etc. In other words, composed. However, this is not backed up by empirical evidence.
Recommendation: 1: strong reject
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 1 poor
Clarity: 2 fair