Title: Fast Rate Bounds for Multi-Task and Meta-Learning with Different Sample Sizes
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multitask learning

generalization bounds

PAC-Bayes

fast-rate

unbalanced
Abstract: We present new fast-rate PAC-Bayesian generalization bounds for multi-task and meta-learning in the unbalanced setting, i.e. when the tasks have training sets of different sizes, as is typically the case in real-world scenarios. Previously, only standard-rate bounds were known for this situation, while fast-rate bounds were limited to the setting where all training sets are of equal size. Our new bounds are numerically computable as well as interpretable, and we demonstrate their flexibility in handling a number of cases where they give stronger guarantees than previous bounds. Besides the bounds themselves, we also make conceptual contributions: we demonstrate that the unbalanced multi-task setting has different statistical properties than the balanced situation, specifically that proofs from the balanced situation do not carry over to the unbalanced setting. Additionally, we shed light on the fact that the unbalanced situation allows two meaningful definitions of multi-task risk, depending on whether all tasks should be considered equally important or if sample-rich tasks should receive more weight than sample-poor ones.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors develop PAC-Bayesian risk bounds for the multi-task and meta-learning settings, in both cases taking care to give bounds for the unbalanced regime (where each task can have a different number of samples) that can give fast rates, even in situations where there exists a task with only a small number of samples. The focus is both on the simple average of tasks and a sample-size-weighted average (task-centric and sample-centric respectively) .The multi-task risk bounds are both of kl-type and Catoni-type, while the meta-learning risk bounds are of kl-type. Along the way, the authors show that the downside of existing (\"standard\") multi-task learning bounds in the unbalanced regime, while also showing how the perhaps-most-straightforward PAC-Bayesian analysis method cannot be useful when there exists a task whose number of samples is very small. The authors lightly touch upon computational issues related to getting the best possible bound from their (implicit) risk bounds. Finally, the authors demonstrate how their bounds fare in practice compared to some standard bounds and a more naive fast rate bound.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nTo my knowledge, prior works haven't shown fast rate PAC-Bayesian risk bounds for settings where tasks have variable sample size (the unbalanced setting). I can see that there is some value in filling in this gap, whether in the multi-task setting or the meta-learning setting. Related to the lack of prior coverage of the unbalanced setting, I don't believe the issue of bounds plagued by $m\\_{\\\\min}$ has previously been raised, and the authors did well in highlighting how a naive approach and some existing bounds can suffer when the minimum sample size (among the training tasks) is very small.\n\nAnother interesting aspect of this work, which unfortunately was severely under-discussed in the main text, is the computational issue of going from the authors' bounds (which are implicit, involving, e.g., Bernoulli kl-divergence terms in the case of the kl-type bounds) to the best possible bound on the multi-task risk (say, the simple average). This issue strikes me as extremely interesting, and the authors apparently have thought about this a lot due to the extra material in the appendix (which, out of respect of the page limit and since it was new content not supporting results appearing in the main text, I regret to say I could not review out of fairness to the other papers in submission to NeurIPS this year). Going forward, I encourage the authors to devote significant attention to this computational aspect in the main text, including at least one figure. This might be where the key technical contribution actually lies for your work. You might even have new theorems as part of this (like, extracting the best possible explicit bound given your existing implicit bounds). In truth, the under-coverage of the computational aspect in the main text is a weakness that can be turned into a major strength.\n\nI think it is nice that the authors made explicit the task-centric and sample-centric objectives. I can't say that this is terribly novel, as task-centric is standard and (without even checking) I feel that sample-centric should have appeared in some works before. Even so, formalizing all of this, along with presenting bounds for each case, is nice.\n\nThe experimental results are encouraging. I have a question about precisely what bound was used for \"Fast rate with $m\\_{\\\\min}$\".\n\n\n## Weaknesses\n\nI felt the paper is quite lacking when it comes to describing how the technical ideas in the present paper relate to previous works. For example, Theorem 2.1 (which I understand the authors do not claim to be a novel result but simply present as a way to provide context for their upcoming, new results) seems to essentially be due to Pentina and Lampert (2014). Yet, the proof in the appendix is written from scratch, without giving credit to previous, related results. \n\nI found to be underwhelming the technical novelty behind the authors' new risk bounds (e.g., Theorem 3.2). Inspection of the proof reveals that the authors follow a standard PAC-Bayesian analysis modulo the application of Lemma A.1; the application of that lemma is similar to how the lemma is used in the work of Zakerinia et al. (2025). I found the lack of a majorly new technical idea to be surprising because the authors say — both in their Introduction and their Section 3 — that their contribution is technical. For a venue like NeurIPS and a paper that positions it as theory paper (as the authors do), please take care to explain what the technical hurdles were.\n\nLess major 1: From the writing, I felt the authors were suggesting that Lotfi et al. (2024) considered PAC-Bayesian bounds. I believe this work only considered compression bounds. It would be better to adjust the writing to avoid sending an incorrect message (please correct me if I'm wrong; I could not find PAC-Bayes bounds in the work of Lotfi et al.).\n\nLess major 2: A somewhat minor presentational aspect is this the authors do not make clear where the proofs of various results happen. For example, a search for \"Theorem 3.2\" in the appendix does not turn up any results. Instead, I eventually found two theorems in the appendix which together provide the two statements of Theorem 3.2.\n\n\n## Additional comments\n\n\nA few comments on the appendix:\n\nLine 495: $\\\\mathcal{R}\\_u$ and $\\\\hat{\\\\mathcal{R}}\\_u$ were never defined, although I could infer their meaning.\n\nLine 506: for completeness you should mention that you're using a geometric grid rather than using a union bound over the infinite set $(1, 4 n m\\_h]$.\n\nEquation (51): Here and in some places below here, the summation should be pulled to the outside; as written, $f\\_i$ is out of scope (since $i$ is \"out of scope\" in the computer-science sense).\n\nLines 527–528: It would help to provide more explanation on precisely which function $f$ you consider (when applying Lemma A.1), what choice of $X\\_i$ you use, etc. I figured this out myself but it took a bit of time as I had to write everything out. Also, note that you are missing an expectation with respect to $f\\_i$ just after the inequality in equation (53).\n\nQUESTIONS:\n1. What would you say is the core technical novelty behind your results? Having looked over some of the proofs, I think it would suffice if you restrict your attention (when answering this question) to the proof of Theorem 3.2\n\n2. Can you point out what bound you are using when you write \"Fast rate with $m\\_{\\\\min}$\" in the experimental results in the main text?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-21"}

comments: SUMMARY:
This paper derives new, fast rates for the multi-task and meta learning problems under imbalanced sample sizes per task. The authors provide results in a PAC-Bayes framework, which situates their contributions within both the MTL/ML and PAC-Bayes literatures. Their results bring to light new phenomena in the behavior of multi-task and meta-learning with imbalances sample sizes.

STRENGTHS AND WEAKNESSES:
Strengths: The paper is exceptionally well-written and clearly motivated. The authors provide an excellent background for the problem under consideration. In particular, Lemma 3.1 demonstrates formally the necessity of the contribution of this work. The main theoretical results are clearly written and genuinely novel. The simulations support the bounds numerically.

Weaknesses: It is not clear that proofs of the theorems are significantly different from existing results on a technical level. I am not an expert in PAC-Bayes, but my reading of the proofs is that they broadly follow the same structure as existing proof techniques. Upon my current reading of the paper, this slightly detracts from the work's novelty, but I look forward to discussing the nuances of the proof with the authors in the discussion section.

QUESTIONS:
1. What are the key technical challenges of the proofs of Theorems 3.2, 3.3, and 4.1? What differentiates the proof techniques used in this paper from existing techniques that are, seemingly, provably suboptimal?

2. My understanding is that the novel behavior that the authors uncover in imbalanced multi-task learning is most prevalent in the task-centric setting. What is the authors' intuition for why task-centric multi-task learning is more affected by imbalanced sample sizes than the sample-centric analog?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-02

comments: SUMMARY:
This paper provides fast risk bounds for multi-task learning and meta-learning, where different tasks could have different sample sizes. It argues that previous bounds cannot be easily generalized to varying sample cases, and argues for a different proof technique required for handling different sample sizes. It also evaluates the bounds numerically on various tasks and show that it improves over the previous bounds.

STRENGTHS AND WEAKNESSES:
The main strength of the paper is the generalization of prior bounds on multi-task and meta-learning. It argues that previous bounds cannot be easily extended to the case of different sample sizes. However, in my opinion, this does not seem true. Looking at bounds of Guan & Lu 2022, the fast bounds for meta learning can be easily generalized to different sample sizes. Using their notation, $\tilde{err}(Q) - \hat{err}(Q)$ in the paper is currently bounded in Proposition 6 in the Appendix. It gives a rate of $1/mn$, assuming n tasks and m samples per task. Instead simply splitting $\tilde{err}(Q) - \hat{err}(Q)$ over different tasks (as in Proposition 3) would give the error rate dependent on each $m_i$ separately.

QUESTIONS:
1. Why can't the proof of Guan & Lu 2022 be generalized to different sample sizes? As stated before, I think in Proposition 6, the error can be split over various tasks separately.

LIMITATIONS:
I think the comparison to previous works and why those cannot be extended to different sample sizes needs to be properly addressed.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 1
DATE: 2025-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work derives PAC-Bayes fast rates for multi-task learning (MTL) with unbalanced datasets.\nThere is prior work on PAC-Bayes for _balanced_ datasets, yet the authors provide a lemma that shows that the usual line of argument fails to go through in the unbalanced setting. \nThus, the authors provide an alternative argument that provides both kl-style and Catoni-style bounds. \nThe prior bounds are both provided in a task-centric (each task is given equal weight) and sample-centric (weight per task is proportional to number of samples for that task) form.\nPrevious fast-rates in the balanced MTL setting are able to be \"numerically inverted\" to provide bounds on the risk. \nThis work studies the non-trivial solution space of such numerical procedures for task-centric MTL.  Finally they conduct experiments to on linear and neural networks that demonstrate the benefit of their approach versus the rate that doesn't scale in aggregate (but with the size of the minimum sample) and the standard rate.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* A study of unbalanced datasets in the PAC-Bayes setting is interesting particularly since it appears to differ from the PAC setting, in which it appears to me that everything goes through with standard arguments in the unbalanced setting. \n* Their study of task-centric and ample centric is natural and valuable. \n* The authors have identified a weakness in the previous arguments for deriving MTL bounds in unbalanced datasets. \n* The authors work recovers prior standard bounds in two senses. First, when all the tasks have the same number of samples we recover the balanced fast-rate. Second, the naive approach that would be dependent on the smallest sample size is unable to recover the standard rate (via the Pinsker's inequality trick).  Given these recoveries of prior work it seems that this is the more correct analysis of MTL learning in general and therefore clarifies the literature to the nuance between balanced and imbalanced datasets. \n* I appreciated the experiments. In particular Figure 1 that shows the benefits of their method as the task becomes easier which shows the benefit of their approach.  \n* The discovery that kl-bounds have difficult numerical inversion in the task-centric version.\n\n**Weakness**\n* In comparison to prior work, the technical innovation appears to be applying Lemma A.1 to each task independently. Then, after applying Lemma A.2 the harmonic mean is intimidate. \n* The difficulty of numerical inversion the task-version appears to limit the applicability of that particular rate (although this may be natural).\n* I appreciate the detailed treatment of meta-learning, yet it appears that once the MTL rates are derived the remaining meta-learning analysis is standard.\n\nQUESTIONS:\n* Is my remark about the meta-learning proofs above representative? \n\n* Reviewing the proof of Lemma 3.1 we see that the issue is that $\\mu$ is unbounded from below. We note that $\\mu$ small is exactly the situation in which we would expect a fact rate to be possible (small empirical error). What if $\\mu$ is small and bounded from below? (i.e., in a non-interpolation setting.) It seems that this removes this weakness of the prior analysis.\n\n* It seems to me that the bound with binomial upper bound (Lemma A.1) facilitates some information sharing between tasks. On one hand, your analysis recovers the balanced MTL setting so it seems your proof is the correct one. On the other hand, it seems that we're losing the information sharing via Lemma A.1 which seems like a bad idea. Is my intuition about the use of Lemma A.1 incorrect? \n\n* Do the authors have a sense of the geometry of the solution space for the numerical computation in Section 3.4? \n\n* Is there anything technically contrasting between task-centric versus sample-centric versions of the proofs? \n\n* Given Corollary 3.4, in comparison to the numerical numerical details in Section 3.4,  it seems that sample-centric is the nicer version of MTL  versus task-centric. Is that the sense of the authors? \n\n* On line 527 shouldn't it be $B(m_i, \\mathcal{R}_i(f_i)/m_i)? As it is doesn't seem to match with  Lemmas A.1-2. This same confusion similarly resides in the application of Lemma A.2 in the final inequality of Equation (53). (Both these also hold for the other proof too. )\n\nLIMITATIONS:\n* Task-centric rates for the risks are hard to bound due to the numerical complexity.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-08"}