Text-to-Model: Text-Conditioned Neural Network Diffusion for Train-Once-for-All Personalization

Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-25 08:19:58

Abstract: Generative artificial intelligence (GenAI) has made significant progress in understanding world knowledge and generating content from human languages across various modalities, like text-to-text large language models, text-to-image stable diffusion, and text-to-video Sora. While in this paper, we investigate the capability of GenAI for text-to-model generation, to see whether GenAI can comprehend hyper-level knowledge embedded within AI itself parameters. Specifically, we study a practical scenario termed train-once-for-all personalization, aiming to generate personalized models for diverse end-users and tasks using text prompts. Inspired by the recent emergence of neural network diffusion, we present Tina, a text-conditioned neural network diffusion for train-once-for-all personalization. Tina leverages a diffusion transformer model conditioned on task descriptions embedded using a CLIP model. Despite the astronomical number of potential personalized tasks (e.g., $1.73\times10^{13}$), by our design, Tina demonstrates remarkable in-distribution and out-of-distribution generalization even trained on small datasets ($\sim 1000$). 
We further verify whether and how \Tina understands world knowledge by analyzing its capabilities under zero-shot/few-shot image prompts, different numbers of personalized classes, prompts of natural language descriptions, and predicting unseen entities.

Main paper: 
 introduction: 
 A large dataset Large language models [1, 5] , like GPT-4, have especially shown emergent intelligence [6] in the knowledge of language through text-totext transformation [7, 8, 1, 5] . Besides, recent progress in text-to-image (e.g., stable diffusion) [9, 4, 2, 10 ] and text-to-video (e.g., Sora) [3, 11] diffusion models has shown the great power of AI in understanding the physical world and generating high-quality images and videos that are virtually indistinguishable from reality [12, 3] . directly generate the model parameters given the human's text prompts to meet the personalization demand of diverse end users.We focus on a practical scenario called train-once-for-all personalization [13] , which means that the generic model is trained just once and can later be customized into a condensed model on the fly for different end-users and requests, given their task descriptions. For example, the CIFAR-100 dataset [14] contains 100 classes, but an end user may just need a personalized model with a certain 10 classes according to a specific scenario (e.g., classifying items in the kitchen). In other words, train-once-for-all personalization targets that train the model once and customize the model to be well performed in a sub-distribution when deployed, and an example is in Figure 1 . But there are tremendous sub-distributions, for the CIFAR-100 example, the number of personalized 10-way tasks is 100 10 = 1.73 × 10 13 , even not taking permutations into consideration, so it is challenging for the GenAI model to generalize. Inspired by recent progress in neural network diffusion [15, 16] , wepropose Tina, a Text-Conditioned Neural Network Diffusion for Train-Once-for-All Personalization.Tina is trained on model parameters with the models' task descriptions, and it can be generalized to unseen tasks, or even unseen classes (entities), given the text prompts.In Tina, a CLIP model [17] is used to embed the users' task descriptions into the diffusion model as the conditions. The diffusion model of Tina is the diffusion transformer (DiT) [12] that is shown to have high expressive power under scaling law in the fields of image [12] and video generation [3] .We demonstrate that DiT's scaling law applies to model parameter generation as well: increasing the number of parameters and data sizes enhances the model's capability to generalize across more challenging tasks that involve scaling the dimension of generated models. However, it is surprising to find that even though the number of personalized tasks is astronomical (e.g., 1.73 × 10 13 for 10-way tasks), by our designs, Tina can generalize on extremely small datasets (∼ 1000 data points) and support different lengths of classification tasks (5-way or 8-way tasks, etc.) in training once. Our analysis shows that Tina can reach both in-distribution and out-of-distribution personalization of generated models. Thanks to the vision-language alignment of CLIP, Tina can also take images as prompts and generalize under few-shot or even zero-shot settings. We also verify whether Tina understands world knowledge by testing its abilities under prompts of natural language descriptions and predicting unseen entities. Our contributions are as follows:• We explore the potential of GenAI in generating personalized models followed by users' text prompts, i.e., text-to-model generation. We open more applications of neural network diffusion; to the best of our knowledge, it is the first paper that takes the text prompts as conditions for neural network diffusion.• We propose Tina, a well-performed text-conditioned neural network diffusion framework for train-once-for-all personalization. Tina can generalize on unseen tasks and entities even given small model datasets.• In addition, we analyze the abilities and the boundaries of Tina and gain insights about whether and how it generalizes and understands world knowledge. 
methodology: 
 2.1 Problem SetupFollowing [13] , we consider image classification for train-once-for-all personalization due to the natural personalization requirements of image classification. We note that our method is not limited to classification tasks and can be extended to other tasks for personalization. Define a task k as classification over a subset of classes Y k ⊂ Y. The goal of personalization is to learn a neural network predictor f θ k : X → Y k , parameterized by θ k . To handle many tasks at the same time, we further assume we have the task description natural text t k for Y k , and it is generally the description of the classes and styles of Y k . We want to build a neural network generator G(t k ) where given t k , it will output the model parameters θ k . Specifically, consider using a large-scale dataset with many classes covering Y to learn the personalized-friendly functionf θ k = G ϕ (t k ) parameterized by ϕ. G ϕis learned on the large dataset to generate any personalized model directly from the task descriptions, and the setup is called train-once-for-all personalization [13] . Train-once-for-all personalization has wide applications in a server-user system, where the model generator G ϕ is learned on the server for personalized cloud services to many future users. We refer to [13] for more detailed advantages and usages of train-once-for-all personalization.Classifier Selection. For a generic network f θ , we consider that it consists of a feature extractor parameterized by ψ with a linear classifier w = [w (1) , . . . , w (|Y|) ] of |Y| vectors for output predictions over all classes in Y. The generic model is trained on the large dataset, and we want to personalize it into a few-way classification task k. One effective method is to build a personalized classifier w k by selecting only the row vectors in w for the relevant classes. Therefore, the personalized model for task k are θ k = {ψ, w k }, and this approach is called classifier selection, which serves as a strong baseline [13] .TAPER. We briefly introduce TAPER [13] proposed by the original paper on train-once-for-all personalization and discuss its limitations. The main idea of TAPER is to train several experts (bases) and learn a mixture network to fuse these experts into a personalized model. It has three stages as follows.• Stage 1: train a generic model on the large dataset.• Stage 2: divide the dataset into several shards and finetune the generic model on each shard respectively for specification. Each finetuned model can be seen as a domain expert.• Stage 3: For a given personalized task, learn an MLP mixer (i.e., the generator G) whose input is the text embedding of the task description and the output is the aggregation weights of the expert models. Then, weighted aggregation is conducted to merge several expert models into a personalized one. Also, the expert models can be finetuned during personalization.TAPER requires finetuning the expert models on the target task, so it is not applicable to unseen tasks without having task-specific data. Also, the MLP mixer only generates the aggregation weights instead of the parameters, so it has limited generalization and expressiveness. While in our design of Tina, we try to construct an end-to-end text-to-model system that can understand the hyper-knowledge residing parameters and can generalize to unseen tasks, even unseen classes.We ii) Out-of-distribution (OOD, the dark green blocks): the tasks are unseen during the generator G's training, and G directly generates the p-Models from the task prompts (the text descriptions). We note that the original TAPER cannot be tested on the OOD tasks since it requires the target personalized training data for finetuning the expert models. To remedy this, we derive TAPER-Mixer to only train the mixer without finetuning the experts and verify its OOD generalization on unseen tasks. # Sample a mini-batch of data 5:(t k , θ k ) ∼ K 6: # Noise p-Model parameters 7: j ∼ U ({1, ..., J}) 8: θ j k ∼ N ( √ ᾱjθk, (1 -ᾱj)I) 9: # Compute the predictions 10: θk ← G ϕ (T (t k ), θ j k , j) 11:# Compute the loss 12:lossEQUATION13: # Update DiT's parameters 14:ϕi+1 ← update(loss; ϕi) 15: end for We use diffusion models as the generative model and follow the main architecture of G.pt [16] that uses a diffusion transformer as the backbone. Analogous to the optimization process that takes random initialization as inputs and outputs the trained models, the diffusion process takes the noise as inputs and gradually denoises to recover the original distributions. Previous works have shown the rationale of neural network diffusion [16, 15, 18] . We choose DiT as the backbone because it can be easily scaled up and is shown to have great generalization and expressiveness. We use signal prediction for the diffusion process and inherit the architecture of GPT-2 [8] as the transformer. The used text encoder is the pretrained ViT-B/32 in CLIP [17] .Training objective. Denote the training set of Tina as K, where each piece of data is a (task description, p-Model) tuple, notated as (t k , θ k ) for task k ∈ K. We denote the CLIP text encoder as T , and given the task description t k , the text embedding is T (t k ). The text encoder is frozen during training.Our DiT model G ϕ takes two vectors as input: the text embedding T (t k ) as conditions and the noised p-Model parameter vector θ j k , where j ∈ [J] denotes the timestep in the diffusion forward noising process. The learning objective of diffusion is to minimize the simplified variational lower bound, which reduces to predicting the denoised p-Model parameters:EQUATIONwhere the timestep j is embedded in DiT by frequency-based encoding [19] . The detailed training procedure is in Algorithm 1. We use DDPM sampling [9] ; add Gaussian noise depicted by the ᾱ to θ k and gradually denoising it.We elaborate the design details of Tina.Parameter tokenization. For a p-Model's parameters θ k , we first flatten all the parameters into a 1-D vector and chunk/tokenize the parameters within each layer. If the chunk size is M and the number of parameters in a certain layer is N , so for this layer, there will be ceil(N/M ) tokens. For some layers smaller than M , the whole layer is a token.Text embedding. Assume the personalized task is a classification task that has c = |Y k | classes. The task description t k is an ordered list of the classes' text descriptions, of which the simplest form is the class entity, e.g., "telephone" and "rabbit". The generated p-Model is expected to have the correct predictions in the same order with t k . In other words, we need Tina to learn the correct classifier orders as the text prompts, which is sequence-to-sequence modeling. Therefore, unlike TAPER, which averages the class embeddings into one, we make every class description as a token by CLIP text encoder and concatenate them in order with position encoding.Encoding and decoding of tokens. We use linear layers as encoders for mapping the parameter tokens and text embedding tokens to the hidden size of DiT. Each token has a different linear layer without weight sharing. The decoders are similar to encoders, which use linear layers, and the encoders transform the transformer's hidden size back to the p-Model's parameter dimension.Between the encoders and decoders, there are transformer attention layers akin to GPT-2.Data augmentation. In [16] , the permutation invariance property [20, 21, 22] is utilized for data augmentation by randomly permuting the neurons without changing the function. However, in our scenario, we find this augmentation will even impede training. We hypothesize that the personalized Parameter inheritance. In [16] , the authors release a pretrained checkpoint of G.pt, which is also DiT for parameter generation. G.pt is pretrained on large datasets of optimization checkpoints;though it has different conditions, designs, and scenarios from ours, we explore whether we can inherit some parameters from the pretrained checkpoints to speed up and boost training. Considering the model sizes and architectures are different, we use a strategy similar to bert2BERT [23, 24, 25] for inheriting parameters.Classification sequence padding. We study how to incorporate more personalized settings where diverse users request for tasks with different numbers of classes. In language models [26, 5] , padding is used to enable sequence-to-sequence learning with different input and output lengths. Inspired by this, we use the padding technique to enable the description-to-classifier sequence of different classification lengths. Specifically, if the user's number of classes is smaller than the maximal length, we pad missing classes with tokens '<->' in the task description list and mask the corresponding classifier vectors with zero-like tensors. We denote this strategy as classification sequence padding, and Tina can learn to adapt to any number of classes within the maximal length.3 Experiments 
experimental setups: 
 Datasets and p-Models. We use three datasets to conduct experiments: Mini-ImageNet [27, 28] ,CIFAR-100 [14] , and Caltech-101 [29] . Mini-ImageNet is a subset of the ImageNet dataset, primarily We use two architectures for personalized models: a simple CNN (dubbed as CNN) and ResNet-20(dubbed as ResNet). The CNN architecture follows [16] , which consists of 2 layers, and the number of parameters is approximately 5K. We take all the parameters of CNN as the input and output of Tina. But for ResNet-20, the number of parameters is nearly 272k, which is too large for Tina's generation. Thus, we explore partial parameter generation following [15] . We only personalize the classifier layers for parameter generation, nearly 640 parameters.For more details about data preparation and p-Models, please refer to Appendix A in the appendix.Compared baselines. We follow the baselines used in the original paper of train-once-for-all personalization [13] . As described in subsection 2.1.3, we use the generic model trained in stage 1 as a baseline, showing the performance without any personalization. Further, we compare the classifier selection method described in subsection 2.1.2, which serves as a strong baseline for personalization [13] . The vanilla TAPER [13] requires finetuning the expert models on the target tasks and cannot generalize on out-of-distribution personalization where only target text descriptions are available. For fair comparisons, we adopt TAPER-Mixer, which adopts the mixer of TAPER for generating the aggregation weights, and the MLP-based mixer can generalize on unseen tasks.Evaluation metrics. For Table 1 , we compare in-distribution personalization and out-of-distribution personalization as elaborated in subsection 2.1.3. For other tables and figures, we report the out-ofdistribution personalization as p-Acc.Hyperparameters. The detailed hyperparameters can be found in subsection A.5 in the appendix.In Table Hidden Sizes of Tina Parameter inheritance. We verify whether Tina can benefit from pretrained parameters. We inherit the parameters from G.pt's [16] checkpoints by the bert2BERT-like method [24] . From Figure 4 (b), it is found that parameter inheritance from pretrained models can help Tina to converge faster, but the final p-Accs are similar.Training images as prompts. In the original design of Tina, the texts are used for the prompts encoded by the CLIP text encoder. We train a Tina with image prompts using CLIP image encoder, and the results are in Figure 4 (c). For each class, we randomly select one single image as the prompts.It is found that text-prompted Tina converges faster than the image-prompted, though the final p-Accs are similar. This is intuitive to understand since texts are known to have higher knowledge density than images [30, 17] , that the class text has richer knowledge representations than a single image. Testing images as prompts. We train text-prompted Tina and verify its zero-shot and few-shot abilities on image prompts, and the results are in Figure 6 How Tina understands world knowledge I: natural language descriptions as prompts. In our implementation of Tina, we adopt a simple prompting that uses the class names as the text prompts.We verify whether Tina actually learns the knowledge in the case where the prompts are replaced by the natural language descriptions at test time. We generate the language descriptions of classes with the assistance of GPT-4 [31] , and we make sure that the descriptions do not include the original class entities. The exemplars are in Table 4 of the appendix. From Figure 6 (c), the results reveal that Tina has zero-shot generalization ability when the prompts are unseen language descriptions, though the p-Accs are lower than the ones of the class-named prompts. It shows that Tina is not just memorizing the class names but also generalizing and understanding the knowledge behind the names and the nuances inherent in the text semantics.How Tina understands world knowledge II: generalization to unseen classes/entities. We divide the CIFAR-100 dataset into two disjoint shards of classes and train a Tina on one shard, then verify its generalization on the unseen classes of another shard. Results in Table 2 showcase that Tina has the intelligence to generalize on unseen classes, while TAPER-Mixer fails when meeting 100% unseen classes. As a generative model, Tina can understand the hyper-level world knowledge embedded in model parameters as well as text semantics and generate models for predicting unseen entities.We make an ablation study for different design choices of Tina. The ablated designs are the ones different from previous literature, such as our design of classifier augmentation, G.pt's design of permutation augmentation [16] , and TAPER's design of merge text embedding as one [13] . The results are in Table 3 . Our classifier augmentation can boost the performance even under small training datasets. Permutation augmentation has negative effects on generating personalized models, and we hypothesize that for Tina's training data, the p-Models finetuned from the same generic model are located in a common loss basin, where permutations will disturb the shared representations.In addition, merging the text embeddings into one will hinder the DiT from learning the sequential classifications, making Tina bad in generalization. 
related works: 
 Diffusion models. Originating from non-equilibrium thermodynamics [32, 33] , diffusion models have evolved significantly. DDPM and DDIM pioneered forward-and-reverse processes in text-toimage generation [9, 34] . Guided-based diffusion models [35] surpassed GAN-based methods in image generation quality. Subsequent models like GLIDE [36] , Imagen [37] , DALL•E 2 [2] , and stable diffusion [4] further advanced image generation and art creation. The diffusion transformer (DiT) [12] introduced a scaling law, with OpenAI's Sora [3] being a notable application in text-tovideo generation, employing DiT architecture at a billion-scale.Parameter generation. Learning to optimize explores neural networks learning update rules for others [38, 39, 40, 41] . Hypernetwork [42] is a meta learning approach that uses networks to modify neural network parameters, differing from our approach of mapping language space directly to parameter space. Hypernetworks are used in federated learning [43] , few-shot learning [44] , and model editing [45] . A concurrent work ModelGPT [46] customizes models by large language models and hypernetworks, while Tina uses conditional neural network diffusion for a different task-train-once-for-all personalization. Neural network diffusion [16, 15] is recently proposed to mimic optimization rules via diffusion for parameter generation, but previous works haven't explored sufficient use cases of such techniques.For more detailed related works (e.g., the works about personalization), please refer to Appendix B. 
discussions: 
 Limitations. Despite the merits of Tina, it has some current limitations. One bottleneck is the input dimension; due to our computation limits, Tina currently supports lightweight models as inputs, and it requires huge computation resources to fully generate large models with millions of parameters. On the one hand, a larger input dimension needs exponentially larger Tina parameters, so more GPUs.On the other hand, a larger input dimension needs more data to converge or generalize, requiring more compute hours. As a remedy, we tried to train a variational autoencoder (VAE) for encoding the p-Model parameters into a low-dimension latent space as in [15] , but the VAE cannot generalize, suggesting more advanced techniques are needed. Another limitation is the generality of Tina, that one single Tina cannot generate personalized models across different sizes and different modalities;in the future, large-scaling pretraining for Tina may be promising to reach this goal.Broader impacts. Tina is the preliminary work of text-to-model generation and will have broader impacts on the machine learning community, especially in the field of generative AI and model personalization. Though in this initial version of Tina, we only showcase its great potential in image classification tasks, Tina is prospective in a wide range of applications and tasks, such as natural language processing, audio recognition, and recommender system. Also, Tina has opened more potential directions for neural network diffusion, and we believe it can inspire more interesting works in the future. 
conclusion: 
 In this paper, we present Tina, a text-to-model neural network diffusion model for train-once-for-all personalization. Tina has shown its great capability in generating personalized models from text prompts, and it can generalize to in-distribution as well as out-of-distribution tasks, zero-shot/few-shot image prompts, natural language prompts, and unseen classes. Tina also supports personalization under different numbers of classes. This paper explores the potential of text-to-model generative AI and opens new applications for neural network diffusion in end-user personalization.meets the dilemma of G.pt, which lacks a specific scenario and use case. Recently, GPD [18] uses the diffusion model for few-shot learning in smart city applications, which showcases the applications of neural network diffusion. However, GPD takes the smart city's knowledge graphs as prompts and is tailored for the specific smart city application that cannot be easily extended to other fields. Our Tina takes language texts as prompts, which is more flexible and can be extended to a wider range of applications for the personalization of user demands.Personalization Instead of training a generic model to provide many users with the same model service, personalization of deep learning models acknowledges users' characteristics and diversity and learns each a customized model. Personalization techniques were introduced in medical AI [49, 50, 51] , recommendation systems [52, 53] , large language models [54, 55] , and especially federated learning [56, 57] . Personalized federated learning studies how to exploit the common knowledge of users and then use it to explore further personalization on users' local datasets under privacy constraints [56] ,and techniques like proximal descent [58, 57] , network decoupling [56, 59] , and clustering [60, 61] are used. Recently, the scenario of train-once-for-all personalization [13] was proposed to bridge the gap between edge-side and server-side personalization. Train-once-for-all personalization aims to utilize server-side computation and generic models for fast and effective personalized adaptation to meet the edge users' demands. The original method TAPER [13] finetunes the generic model into several base models and learns MLP-based hypernetworks as mixers to fuse the base models into the personalized one given users' task descriptions. However, the MLP mixer has limited generalization capability, and it cannot be applied to unseen tasks, whereas our Tina learns the text-to-model world knowledge and can be generalized to out-of-distribution samples, modalities, and domains.• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: See Appendix A for details.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See section 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: NA Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: As it is.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Justification: As it is.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper suggested a new approach in learning model parameters for new tasks, by training a Diffusion model on text input. By describing the task in natural language, the model generates parameter weights. The current version of submission requires to address the following major concerns,

1- `scalability of experimentation`: The model parameter size in the experiments is too small; larger models are needed to evaluate effectiveness.

2- `statistical analysis`: lack of statistical results in the Exp section, eg, Table 1, Table 2.

Reviews: review: paper_summary: paper_summary: This work introduces Tina, a text-conditioned neural network diffusion model designed for generating personalized models from text prompts. Tina aims to enable efficient personalization by training a generic model once and then customizing it for various end-user tasks using task descriptions. Leveraging a diffusion transformer model and CLIP-based text embeddings, Tina demonstrates the ability to generate models for a wide range of personalized tasks. The approach shows promising results in generalizing to both seen and unseen tasks, achieving state-of-the-art performance in several benchmarks.
strength_weakness: strength_weakness: 1. Some methodological details are sparse, such as the specific configurations and hyperparameters used for training Tina. Providing more granular details could help readers replicate the experiments.
2. The reason for adopting DiT as the weight generation model is not well justified. It would be good to see some results of adopting different kinds of diffusion models.
questions: questions: 1. It would be good to clarify how exactly the parameters are inherited from G.pt for initialization.
2. Tina encodes and decodes the parameter tokens with linear layers, which are trained along with the DiT model in an end-to-end manner. I'm curious about what are the benefits of this one-stage manner compared to the original neural network diffusion, which trains an autoencoder and diffusion model separately. And what would the performance be if you adopted the same pipeline of neural network diffusion for weight generation?
limitations: limitations: While limitations are discussed, the manuscript could benefit from a discussion of the scalability of Tina to larger datasets and more complex tasks.
paper_summary: paper_summary: To generate personalized models for a variety of end-users and tasks via text prompts, this paper introduces Tina, a text-conditioned neural network diffusion model. Tina employs a diffusion transformer model, complemented by a CLIP model to embed task descriptions. Remarkably, Tina demonstrates superior generalization capabilities even on small-scale datasets, performing well both within and outside the distribution of the training data. Furthermore, Tina exhibits robust performance under zero-shot/few-shot image prompts, natural language instructions, and unseen categories.
strength_weakness: strength_weakness: 1.It is better to includes more comprehensive and competitive baselines to show the model’s effectiveness and advance. The two baselines come from one paper published in 2023. As for the experimental setting involves three widely-used datasets, I am wondering whether the experimental results excels or perform similarly to the SOTA performance on some of the three datasets. In other word, is it possible to apply the proposed strategy to some more advanced framework to make the performance similar to the SOTA, which ensure the proposed method have real applications in the real use.

2.The base model is CNN or ResNet in the experiments. Is the proposed method generalized to more advanced framework? Applying the proposed method on more advanced framework and obtain more advance performance indicates that the method has potential to be used in the real life.

3.We suggest providing necessary explanations in the captions of the model framework overview.
questions: questions: None
limitations: limitations: The limitations are fine with me.
paper_summary: paper_summary: The paper introduces Tina, a text-conditioned neural network diffusion model designed for train-once-for-all personalization. Tina utilizes a diffusion transformer model conditioned on task descriptions embedded using a CLIP model. This innovative approach aims to generate personalized models for various end-users and tasks based on text prompts, demonstrating significant generalization capabilities even when trained on relatively small datasets (~1000 samples). The model is evaluated under zero-shot/few-shot image prompts, varying numbers of personalized classes, natural language descriptions, and predicting unseen entities to assess its understanding of world knowledge.
strength_weakness: strength_weakness: - The model parameter size in the experiments is too small; larger models are needed to evaluate effectiveness.
- In Table 1, the results of direct fine-tuning should be included.
- We might need an ablation study on the impact of text prompts.
- We might need an ablation study to determine if the model merely memorizes and reproduces parameters.
- Figure 2 requires polishing for better clarity.
questions: questions: Can you provide your prompts for Appendix A.2?
If the authors solve my questions, I will raise my scores.
limitations: limitations: The model size is too small in exp.

Comments: title: Thanks for your feedback
Thanks for the post-rebuttal response
Thanks for the post-rebuttal response
Thanks for responding my comments
comment: We thank the reviewer for the post-rebuttal feedback. Here, we provide the following response to address your remaining concerns.

- We may kindly remind that due to NeurIPS’s policy, during rebuttal, we cannot edit and revise the submission (the paper PDF), hoping you could understand. The added results during the rebuttal are in the PDF of the [General Response](https://openreview.net/forum?id=nblJSwtdrJ&noteId=zVKd9OZqeO). We kindly promise that all the issues raised by the reviewers and all the results presented in the rebuttal will be included in the future version of the paper once accepted.
- For the model size, actually, we have already included the details in the original submission, i.e., in Lines 240-245. Knowing your concern about the model size, during rebuttal, we have conducted experiments on ViT-B/32 (pretrained by CLIP), whose size is 78M (millions), much larger than CNN and ResNet, and the results are promising. We will take your advice on putting the details of model sizes in the Table captions of the paper.
- For the statistical results, actually, the results in our paper (including the reviewer mentioned Tables 1 and 2) are statistical and fair since, for every setting, we test the personalization performances across over 100 tasks and take the average scores presented in the tables, and each task includes more than 1000 testing image samples. We believe this evaluation measurement is statistical, representative, and fair for the compared methods. Sorry for the confusion; we will include these evaluation details in the paper.

We appreciate the opportunity to address your further concerns, and we sincerely apologize for any confusion and misunderstanding caused before. Your support is important to our paper and we deeply appreciate your valuable comments and feedback. If our response has relieved your concerns, we sincerely hope that you might consider increasing your score. This would be extremely important to us, and we hope for your understanding. 

Thanks again for your efforts and time in reviewing our paper.
Thanks for your response and additional results.

1. I am still concerned about the model size. You can directly include model size parameter size in the Table and paper

2. The author might need to include statistical results in the Exp section, eg, Table 1, Table 2.

3. Please include the revised Figure and results in the final version.

Overall, I will keep my score.
Many thanks for your post-rebuttal feedback. We are deeply grateful for your efforts in engaging with our work and your support in raising your score. Thanks again for your time and attention.
Thank you for the thorough and thoughtful rebuttal. The authors have successfully addressed my concerns, so I am increasing my rating from 5 to 6.
Many thanks for your time and efforts in reviewing our work. We greatly appreciate your recognition of our work and are pleased to hear that we have addressed your concerns.
I have checked the authors' rebuttal and reviewers' comments. The authors' rebuttal sounds fine with me, which addressed most of my concerns. The authors add some additional experiements (i.e. on ViT) for my concerns as well as other reviewers' questions. 

I still would like to vote for this paper. Considering other reviewers' comments and the overall quality of this paper, I tend to keep my score.