SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Train ’n Trade: Foundations of Parameter Markets
Tzu-Heng Huang, Harit Vishwakarma, Frederic Sala
{thuang273, hvishwakarma}@wisc.edu,fredsala@cs.wisc.edu
Organizations typically train large models individually. This is costly and time-consuming, particularly for large-scale foundation models. Such vertical production is known to be suboptimal. Inspired by this economic insight, we ask whether it is possible to leverage others' expertise by trading the constituent parts in models, i.e., sets of weights, as if they were market commodities. While recent advances in aligning and interpolating models suggest that doing so may be possible, a number of fundamental questions must be answered to create viable parameter markets. In this work, we address these basic questions, propose a framework containing the infrastructure necessary for market operations to take place, study strategies for exchanging parameters, and offer means for agents to monetize parameters. Excitingly, compared to agents who train siloed models from scratch, we show that it is possible to mutually gain by using the market, even in competitive settings. This suggests that the notion of parameter markets may be a useful paradigm for improving large-scale model training in the future.
Costs to build powerful state-of-the-art machine learning models, such as foundation models (e.g., GPT-3 [1], T5 [2], PaLM [3], BLOOM-176B [4] and OPT-175B [5]), have increased enormously. These costs can easily reach millions of dollars and even exceed that amount. For example, training the 11 billion-parameter T5 model is estimated to take around 1.3 million dollars for a single training run [6]. Unfortunately, few organizations and fewer individuals are sufficiently well-capitalized to afford such training costs.
One approach to reduce expense is to broadly distribute training workloads, such as in decentralized training [7,8,9,10]. However, this is limiting; even in the decentralized setting, participants must first agree to train a shared model and at least minimally coordinate the training process. For this reason, such techniques cannot be applied when organizations develop different models for different purposes on different timelines. In these scenarios—the most common solution in large-scale model development—models are trained individually regardless of high cost.
A natural question is whether such vertical production can be broken down into parts that can be more easily built, re-used, and exchanged. Taking inspiration from other areas of manufacturing, we observe that most products are not built in vertical silos, but from components that are traded in markets. Economic agents, even when competing against each other, buy, sell, and trade such components to leverage the expertise of other agents so that production costs can be lowered.
This leads us to ask whether subsets of trained weights can be thought of as constituent parts to be bought and sold on parameter markets. Such markets may provide mutual benefits for both buyers and sellers. Buyers are able to purchase well-trained parameter sets directly as commodities to leverage the training expertise of others and then use them to improve model performance. Sellers (i.e., owners of partially or fully-trained models) are able to monetize parameters as a second profit center, in addition to the downstream activity enabled by using their models.
Challenges and Proposed Approach. How can we build and use such markets? To answer this, we must first overcome several obstacles. An immediate challenge is the notion of alignment: models trained in isolation may not have parameter sets that correspond in any natural way, especially if these models have differing purposes. Excitingly, recent work suggests that it is possible to align model components and then merge them via linear interpolation [11, 12, 13, 14, 15, 16, 17]. Similarly, it is known that training data can be potentially recovered from weights or gradients so privacy is an additional challenge [18, 19, 20]. We tackle the orthogonal question:
If model alignment and sufficient privacy can indeed be assured, how can a viable parameter marketplace be designed?
There are three fundamental challenges in doing so:
1. How should agents (users in the market) decide to perform transactions? Discovering and verifying useful model parameter sets on the market without prior knowledge is challenging for buyers. To address this issue, we introduce a trusted third-party organization, the broker. The broker enables a "try-before-purchase" mechanism for buyers to examine the quality of parameters. This is a common approach in the existing works on data marketplaces [21, 22] as it allows buyers to evaluate the quality of data before purchase. Doing so with parameters rather than data presents additional complications that must be resolved.
2. What rewards may agents gain? An important consideration when using such a framework is whether agents can expect to see any gains. Indeed, if the process of exchanging and validating parameter sets does not yield any improvements when compared to siloed training, there is no reason to participate in parameter markets. We provide theoretical and empirical analyses validating the improvements gained from participating in such markets.
3. How to monetize model parameters in a competitive market? In settings where parameters are bought and sold, rather than bartered, it can be challenging to price these assets. Both the seller and the buyer may not have a clear understanding of the other's valuation of the parameters, which makes it difficult for each to maximize their revenues in a trade. To address this issue, we apply a Bayesian-optimal pricing mechanism [23] to provide valuation of parameter sets and study Nash bargaining [24] to find market prices in negotiation.
Results and Contributions. We propose and formulate a viable marketplace to trade parameters for machine learning models. We validate it in both theoretical and empirical settings. Theoretically, in basic scenarios we show how agent training converges faster through purchasing parameters in the market. We offer bounds on the improvement gained via trading when training linear models. Empirically, we conduct experiments in a variety of practical scenarios to validate the framework's effectiveness. We demonstrate that compared to agents who stay outside the market and train models in isolation, participating in parameter trading, even trading subsets of the full set of model parameters, provides benefits to efficient model training and better model performance. For example, when training and trading parameters of ResNet20 on TinyImageNet, two agents improve their performance by gaining accuracy improvements of  $+10.02\%$  and  $+15.93\%$  versus separate training. We also demonstrate the success of price estimation to monetize parameters and negotiate prices.
First, we describe two related concepts: data and model—as opposed to parameter—marketplaces. We then give background on model alignment techniques, which are used in our framework.
Data Marketplaces. Data is a key ingredient in machine learning pipelines. There is a rich vein of work proposing infrastructure to trade data as a commodity [21, 22, 25, 26, 27, 28, 29]. Such marketplaces have also been instantiated in industry, including in services such as Amazon Web Services (AWS) Data Exchange, Microsoft's Azure Data Marketplace, and Google's Cloud Data Catalog. Such research; however, cannot be directly applied to trading parameters. It is relatively easy to evaluate the valuation of a data trade using basic statistical measurements. In contrast, the value of parameters is challenging to measure, as it can only be determined after testing and depends on the model's performance.
Model Marketplaces. There have been several efforts to build markets to trade entire model instances trained by a centralized broker [30, 31]. Two major obstacles that need to be overcome are determining the value and pricing models, and safeguarding privacy. To address the former issue, a noise-injection mechanism has been suggested, which involves assessing the accuracy of an optimal model with random Gaussian noise to determine its worth and creating a price-error curve for selling it on the market [30]. The latter issue has been tackled by proposing a system that apply differential privacy while still maximizing revenue [31]. In contrast to trading entire model instances for downstream use, parameter markets are far more refined, enabling each user in the market to train their own models for their own purposes while gaining from others' training runs.
Model Alignment. Models that are trained with different batch orders or initialization weights may not be properly aligned. Directly merging purchased parameters through interpolation may fail. The process of aligning parameters in model training is therefore critical. Recent studies have explored the geometric relationship between models [11, 12, 13, 14, 15] and proposed methods for aligning two sets of neural network parameters [16, 17]. We use such techniques as a building block in our proposed market infrastructure. Through proper model alignments, we expect agents are able to find and purchase desired parameter sets in the market.
3 A Framework for Parameter Markets
We provide a general description of the proposed marketplace and then discuss each component in depth.
3.1 General Marketplace Framework
Figure 1 depicts a two-agent version of the marketplace. Multiple agents training models for potentially different tasks seek to buy or sell sets of parameters. Buying well-trained parameter sets enables reaching a goal performance faster while selling such parameters produces profits.
A high-level description of the trading process follows. First, agents send their own parameters to the market ①. A third party (the broker) operates a "try-before-purchase" routine to align and merge parameters for buyers ②. The broker privately informs agents of the gains or losses resulting from a potential trade using validation data ③. Based on this information, a buyer values a seller's parameters, and then makes a trading decision. If the buyer is willing to purchase, a quote request is sent out, and both sides generate and report their valuations to the broker ④, ⑤. The broker helps both parties negotiate the price until they reach an agreement ⑥. Afterwards, the broker ships parameters to the buyer and transfers the payment to the seller, completing the trade.
Figure 1: Overall workflow in a two-agent market. Blue and orange blocks represent actions taken by agents and the broker, respectively. In this example, agent  $A$  is informed of a potential gain through purchasing agent  $B$ 's parameters. Hence, agent  $A$  sends a quotation request to inquire about purchasing parameters. Then, broker helps both sides negotiate on the price of agent  $B$ 's parameters.
3.2 Market Setup
In the following sections, we discuss each foundational concept in parameter markets. We first fix some notation. For agent  $u$ , let  $D_{u} = \{s_{u,i}\}_{i=1}^{n_{u}}$  be the samples drawn from a distribution  $\mathcal{D}^u$  supported on  $S$ . At round  $t$ , let  $\theta_{u}^{t} \in \mathbb{R}^{d}$  be trained parameters that agent  $u$  has access to, and let  $\hat{\mathcal{L}}_{u}(\theta_{u}^{t})$  be the empirical loss of agent  $u$  for the corresponding model measured on data  $D_{u}$ . The
empirical loss is defined with the agents' loss function  $\ell$  in a standard fashion,
$$
\hat {\mathcal {L}} _ {u} \left(\theta_ {u} ^ {t}\right) := \frac {1}{n _ {u}} \sum_ {i = 1} ^ {n _ {u}} \ell \left(\theta_ {u} ^ {t}, s _ {u, i}\right).
$$
While our framework can handle general settings (i.e., any well-trained parameter sets can be traded), we focus on supervised learning for ease of exposition. We assume  $\mathcal{S} = \mathcal{X} \times \mathcal{Y}$  where  $\mathcal{X}$  and  $\mathcal{Y}$  are the instance and label spaces respectively. Thus,  $s_{u,i} = (x_{u,i},y_{u,i})$ , and also denote  $D_{u}$  as  $(X_{u},Y_{u})$  where  $X_{u}$  is the set of points  $x_{u,i}$  and  $Y_{u}$  is the set of corresponding labels. We drop the superscript  $u$  when it is otherwise clear from the context.
Agents and Broker. For simplicity, suppose there are two agents  $A, B$  in the market. Agent  $A$  has  $n_a$  data points  $(X_a, Y_a)$ , and agent  $B$  has  $n_b$  data points  $(X_b, Y_b)$ . There is also a trusted third party, the broker  $Z$ , helping the two agents operate in the market impartially. The broker has access to a validation dataset  $(X_z, Y_z)$  of size  $n_z$  with noiseless samples from both distributions.
In the proposed trading framework, the two agents train their models locally first and then trade with each other. At the beginning of round  $t$ , agents perform a standard gradient descent update using their own datasets. That is,  $\hat{\theta}_u^t \coloneqq \theta_u^{t-1} - \eta \nabla \hat{\mathcal{L}}_u(\theta_u^{t-1}), u \in \{a, b\}$ , where  $\eta$  indicates the step size.
Using Purchased Parameters. In the market, parameters can be bought or sold. To use acquired parameters, several model alignment techniques can be employed [16, 17]. For a potential trade, broker aligns seller's parameter sets to match a buyer's. Post-alignment, the parameters can be merged as a straightforward convex combination. For instance, if agent  $A$  is willing to buy parameters from agent  $B$ , and a trade occurs eventually, the post-alignment combination will have the form  $\bar{\theta}_a^t \coloneqq (1 - \alpha)\dot{\theta}_a^t + \alpha\dot{\theta}_b^t$ ,  $\alpha \in (0,1]$ . Here  $\alpha$  and (for agent  $B$ ,  $\beta$ ) are weights indicating the portion of trained parameters from the seller side in the merged parameters.
3.3 Try-Before-Purchase Mechanism
However, before making any trading decision, agents are unaware of the potential benefits they can achieve through purchasing. Our proposal is for a neutral broker to implement a "try-before-purchase" mechanism which assists agents. It does so by helping them align and merge parameters then pre-evaluate their quality by using the broker's dataset  $(X_{z},Y_{z})$ . In the try-before-purchase mechanism, the broker can pick the optimized weights  $\alpha$  or  $\beta$  for buyer agents by minimizing the empirical loss. We write  $\alpha \coloneqq \arg \min_{\nu \in (0,1]}\hat{\mathcal{L}}_z\big((1 - \nu)\dot{\theta}_a^t +\nu \dot{\theta}_b^t\big)$  and  $\beta \coloneqq \arg \min_{\nu \in (0,1]}\hat{\mathcal{L}}_z\big((1 - \nu)\dot{\theta}_b^t +\nu \dot{\theta}_a^t\big)$ .
Using the optimized purchased weights, the broker calculates and communicates to agents their gain or loss from the trade in a confidential manner. We denote the gain-from-trade for agent  $u$  by  $\Delta_u^t$ ,  $u \in \{a, b\}$ , which serves as prior knowledge to help agents make informed decisions about whether to make purchases or not. Generally, the notion of gain-from-trade is to compare relative improvement versus not trading. The trading benefits can be expressed in various ways, such as the difference between agent's loss before and after the trade. For example, it might take the form  $\Delta_u^t = \hat{\mathcal{L}}_z(\dot{\theta}_u^t) - \hat{\mathcal{L}}_z(\bar{\theta}_u^t)$ . Other ways to define the gain-from-trade include using the relative improvement ratio on the empirical loss or improvement ratio on the (estimated) parameter recovery error.
If the gain-from-trade  $\Delta_u^t$  does not indicate any benefit for buyer agent  $u$ , the agent will not be willing to make a purchase. Consequently, no quote request will be sent out, and no trade will take place. In such a scenario, the parameters  $\dot{\theta}_u^t$  will remain with agent  $u$  until the next round of gradient descent. The final parameters at the end of round  $t$ , denoted by  $\theta_u^t$ , can be either  $\bar{\theta}_u^t$  or  $\dot{\theta}_u^t$ . To indicate a trade, we use the indicator variable  $I_u^t$ :
$$
\theta_ {u} ^ {t} := \left(1 - I _ {u} ^ {t}\right) \cdot \dot {\theta} _ {u} ^ {t} + I _ {u} ^ {t} \cdot \bar {\theta} _ {u} ^ {t} \quad \text {w h e r e} I _ {u} ^ {t} = \left\{ \begin{array}{l l} 1 & \text {i f a g e n t u b u y s p a r a m e t e r s ,} \\ 0 & \text {i f a g e n t u d o e s n o t m a k e a p u r c h a s e .} \end{array} \right.
$$
3.4 Valuation and Pricing Mechanism
Once the gain-from-trade has been determined and trading decisions have been made, quote requests with purchased weights are circulated throughout the market. Both buyers and sellers begin to assess various parameters in order to negotiate prices. Each agent has their own private valuation function, denoted by  $v_{u}:\mathbb{R}^{d}\mapsto \mathbb{R}^{+}$ , which quantifies their trading benefits produced by specific parameters  $\theta \in \mathbb{R}^d$ . The valuation function of each agent is private and is not exposed to others. For instance, agent  $A$ 's valuation function is denoted by  $v_{a}(\dot{\theta}_{b}^{t})$  for the value of agent  $B$ 's parameters for purchasing, and  $v_{a}(\dot{\theta}_{a}^{t})$  is the value of their own parameters for selling. In order to generate revenue from a trade,  $v_{a}(\dot{\theta}_{b}^{t})$  is the highest price that agent  $A$  is willing to bid for purchase, while  $v_{a}(\dot{\theta}_{a}^{t})$  is the lowest price that agent  $A$  is willing to ask for in order to sell.
Once valuations are completed, the broker—in their role as an impartial middleman—assists in bargaining the market price to maximize revenue for both sides. The negotiation process for prices continues until a mutually acceptable market price for parameters is reached. When a buyer offers to pay a lower price for certain parameters than the seller is willing to accept, the trade cannot be fulfilled. To analyze this negotiation process, we treat it as a Nash bargaining problem [24]. We assume that the broker is knowledgeable about the valuations of both parties (not the valuation functions themselves) and sets the price by maximizing the revenue of both agents impartially. An agent's revenue, defined as  $U_{u}$ , is derived from two sources: the profit earned from selling self-parameters and the profit gained from buying parameters. We employ the popular Cobb-Douglas function to maximize both agents' revenue [32]. We denote the market price for agent  $u$ 's parameters at round  $t$  as  $P_{u}^{t} \in R^{+}$ , then formulate the problem accordingly. Set  $U_{a}(P_{a}^{t}, P_{b}^{t}) := (P_{a}^{t} - v_{a}(\dot{\theta}_{a}^{t})) + (v_{a}(\dot{\theta}_{b}^{t}) - P_{b}^{t})$  and  $U_{b}(P_{b}^{t}, P_{a}^{t}) := (P_{b}^{t} - v_{b}(\dot{\theta}_{b}^{t})) + (v_{b}(\dot{\theta}_{a}^{t}) - P_{a}^{t})$ . Then we have the problem
$$
\operatorname * {a r g m a x} _ {P _ {a} ^ {t}, P _ {b} ^ {t}} U _ {a} (P _ {a} ^ {t}, P _ {b} ^ {t}) \times U _ {b} (P _ {b} ^ {t}, P _ {a} ^ {t})
$$
$$
\text {s . t .} \quad P _ {a} ^ {t} \in \left[ v _ {a} \left(\dot {\theta} _ {a} ^ {t}\right), v _ {b} \left(\dot {\theta} _ {a} ^ {t}\right) \right], \quad P _ {b} ^ {t} \in \left[ v _ {b} \left(\dot {\theta} _ {b} ^ {t}\right), v _ {a} \left(\dot {\theta} _ {b} ^ {t}\right) \right].
$$
By solving this problem, the broker can determine the difference in price, denoted by  $\Delta P_{ab}^{t}$ , using
$$
\Delta P _ {a b} ^ {t} = P _ {a} ^ {t} - P _ {b} ^ {t} = \frac {1}{2} \left(v _ {b} \left(\dot {\theta} _ {a} ^ {t}\right) + v _ {a} \left(\dot {\theta} _ {a} ^ {t}\right) - v _ {a} \left(\dot {\theta} _ {b} ^ {t}\right) - v _ {b} \left(\dot {\theta} _ {b} ^ {t}\right)\right). \tag {1}
$$
The steps for solving this problem are shown in the Appendix A. The resulting price difference  $\Delta P_{ab}^{t}$  represents the amount of money that needs to be transferred between parties.
4 Instantiating the Market: Concrete Examples
Now we give a concrete instantiation of the market that we described in the previous section.
Valuations. There are various ways for agents to define valuation functions (i.e. based on agent's preference, training budget, or model performance). Here we assume that in the market, agents to purchase use gain-from-trade  $\Delta_{u}^{t}$ , which can be seen as a notion of relative performance improvement, to assess the value of parameters so that  $v_{u}(\dot{\theta}_{u^{\prime}}^{t}) = \Delta_{u}^{t}$ , where  $u^{\prime}$  represents their seller agent.
However, assessing the value of self-parameters for agent  $u$ , who is a seller, is a difficult task as there is no clear information available from the broker regarding the quality of such parameters. The best approach for a seller to maximize profit is to set a price as close as possible to the buyer's valuation, which is the highest price that the buyer is willing to pay. To arrive at this virtual valuation, we use the Bayesian-optimal pricing mechanism described in Myerson's work [23]. This mechanism enables the seller to monetize self-parameters. Under this Bayesian mechanism, we assume that the seller is also aware that the buyer's valuation arises from gain-from-trade, and that their valuation function is derived from a known probability distribution. We discuss these common priors in Appendix G.
Suppose the buyer's valuation has a cumulative distribution function  $F_{v}$ . If the seller sets a price of  $P$ , the probability that the buyer is willing to purchase is  $\mathbb{P}(P < v) = 1 - \mathbb{P}(v \leq P) = 1 - F_{v}(P)$ . The expected revenue for the seller is  $P \times (1 - F_{v}(P))$ . Hence, the optimal price to ask for can be
Algorithm 1 Single Round of Parameter Trading
Input:  $(X_{a},Y_{a}),(X_{b},Y_{b}),(X_{z},Y_{z}),\theta^{*},\theta_{a}^{t - 1},\theta_{b}^{t - 1}$    
Output:  $\theta_b^t$ $\dot{\theta}_u^t\gets \theta_u^{t - 1} - \eta \nabla \hat{\mathcal{L}}_u(\theta_u^{t - 1}),u\in \{a,b\}$  ▷agents' local training   
 $\bar{\theta}_{a}^{t} = (1 - \alpha)\dot{\theta}_{a}^{t} + \alpha \dot{\theta}_{b}^{t},\quad \alpha = \arg \min_{\nu \in (0,1]}\hat{\mathcal{L}}_{z}\big((1 - \nu)\dot{\theta}_{a}^{t} + \nu \dot{\theta}_{b}^{t}\big)$  ▷broker's try-before-purchase   
 $\bar{\theta}_{b}^{t} = (1 - \beta)\dot{\theta}_{b}^{t} + \beta \dot{\theta}_{a}^{t},\quad \beta = \arg \min_{\nu \in (0,1]}\hat{\mathcal{L}}_{z}\big((1 - \nu)\dot{\theta}_{b}^{t} + \nu \dot{\theta}_{a}^{t}\big)$  ▷broker's try-before-purchase   
 $\Delta_u^t = \frac{\|\dot{\theta}_u^t - \theta^*\|_2^2}{\|\bar{\theta}_u^t - \theta^*\|_2^2},u\in \{a,b\}$  ▷inform agents about gain-from-trade   
if  $\Delta_b^t >1$  then ▷agent  $B$  sends a quotation request with  $\beta$    
agent  $A,B$  provide valuations to broker ▷agent  $A$  's valuation is estimated by the bounds of  $\Delta_b^t$    
if  $v_{b}(\dot{\theta}_{a}^{t})\geq v_{a}(\dot{\theta}_{a}^{t})$  then ▷ship merged parameters   
else   
return  $\dot{\theta}_b^t$  ▷negotiation fails   
else   
return  $\dot{\theta}_b^t$  ▷agent  $B$  decides not to buy
found by maximizing the expected revenue, and it satisfies
$$
P ^ {*} := \frac {1 - F _ {v} \left(P ^ {*}\right)}{F _ {v} ^ {\prime} \left(P ^ {*}\right)}. \tag {2}
$$
Linear Model Case Study. To further illustrate a concrete example of the seller's virtual valuation, we consider pricing parameters for training linear models. For simplicity, we assume that the broker knows the true parameters  $\theta^{*}$  though this is not necessary in practice, as these can be approximated using the broker's validation dataset. Both agents' data is sampled from the same distribution so that true parameters are identical. Additionally, we take the gain-from-trade revealed by broker to be the ratio of squared parameter estimation error. We write
$$
\Delta_ {u} ^ {t} := \frac {\| \dot {\theta} _ {u} ^ {t} - \theta^ {*} \| _ {2} ^ {2}}{\| \bar {\theta} _ {u} ^ {t} - \theta^ {*} \| _ {2} ^ {2}}. \tag {3}
$$