Title: Train 'n Trade: Foundations of Parameter Markets
Venue: NeurIPS 2023
Accepted: yes
Keywords: Parameter Market

Pricing

Efficient Model Training
Abstract: Organizations typically train large models individually. This is costly and time-consuming, particularly for large-scale foundation models. Such vertical production is known to be suboptimal. Inspired by this economic insight, we ask whether it is possible to leverage others' expertise by trading the constituent parts in models, i.e., sets of weights, as if they were market commodities. While recent advances in aligning and interpolating models suggest that doing so may be possible, a number of fundamental questions must be answered to create viable parameter markets. In this work, we address these basic questions, propose a framework containing the infrastructure necessary for market operations to take place, study strategies for exchanging parameters, and offer means for agents to monetize parameters. Excitingly, compared to agents who train siloed models from scratch, we show that it is possible to mutually gain by using the market, even in competitive settings. This suggests that the notion of parameter markets may be a useful paradigm for improving large-scale model training in the future.

Train 'n Trade: Foundations of Parameter Markets

Abstract

Organizations typically train large models individually. This is costly and time-consuming, particularly for large-scale foundation models. Such vertical production is known to be suboptimal. Inspired by this economic insight, we ask whether it is possible to leverage others' expertise by trading the constituent parts in models, i.e., sets of weights, as if they were market commodities. While recent advances in aligning and interpolating models suggest that doing so may be possible, a number of fundamental questions must be answered to create viable parameter markets. In this work, we address these basic questions, propose a framework containing the infrastructure necessary for market operations to take place, study strategies for exchanging parameters, and offer means for agents to monetize parameters. Excitingly, compared to agents who train siloed models from scratch, we show that it is possible to mutually gain by using the market, even in competitive settings. This suggests that the notion of parameter markets may be a useful paradigm for improving large-scale model training in the future.

1 Introduction

Costs to build powerful state-of-the-art machine learning models, such as foundation models (e.g., GPT-3 [1], T5 [2], PaLM [3], BLOOM-176B [4] and OPT-175B [5]), have increased enormously. These costs can easily reach millions of dollars and even exceed that amount. For example, training the 11 billion-parameter T5 model is estimated to take around 1.3 million dollars for a single training run [6]. Unfortunately, few organizations and fewer individuals are sufficiently well-capitalized to afford such training costs.

One approach to reduce expense is to broadly distribute training workloads, such as in decentralized training [7, 8, 9, 10]. However, this is limiting; even in the decentralized setting, participants must first agree to train a shared model and at least minimally coordinate the training process. For this reason, such techniques cannot be applied when organizations develop different models for different purposes on different timelines. In these scenarios-the most common solution in large-scale model development-models are trained individually regardless of high cost.

A natural question is whether such vertical production can be broken down into parts that can be more easily built, re-used, and exchanged. Taking inspiration from other areas of manufacturing, we observe that most products are not built in vertical silos, but from components that are traded in markets. Economic agents, even when competing against each other, buy, sell, and trade such components to leverage the expertise of other agents so that production costs can be lowered.

This leads us to ask whether subsets of trained weights can be thought of as constituent parts to be bought and sold on parameter markets . Such markets may provide mutual benefits for both buyers and sellers. Buyers are able to purchase well-trained parameter sets directly as commodities to leverage the training expertise of others and then use them to improve model performance. Sellers (i.e., owners of partially or fully-trained models) are able to monetize parameters as a second profit center, in addition to the downstream activity enabled by using their models.

Challenges and Proposed Approach. How can we build and use such markets? To answer this, we must first overcome several obstacles. An immediate challenge is the notion of alignment : models trained in isolation may not have parameter sets that correspond in any natural way, especially if these models have differing purposes. Excitingly, recent work suggests that it is possible to align model components and then merge them via linear interpolation [11, 12, 13, 14, 15, 16, 17]. Similarly, it is known that training data can be potentially recovered from weights or gradients so privacy is an additional challenge [18, 19, 20]. We tackle the orthogonal question:

If model alignment and sufficient privacy can indeed be assured, how can a viable parameter marketplace be designed?

There are three fundamental challenges in doing so:

How should agents (users in the market) decide to perform transactions? Discovering and verifying useful model parameter sets on the market without prior knowledge is challenging for buyers. To address this issue, we introduce a trusted third-party organization, the broker . The broker enables a "try-before-purchase" mechanism for buyers to examine the quality of parameters. This is a common approach in the existing works on data marketplaces [21, 22] as it allows buyers to evaluate the quality of data before purchase. Doing so with parameters rather than data presents additional complications that must be resolved.

What rewards may agents gain? An important consideration when using such a framework is whether agents can expect to see any gains . Indeed, if the process of exchanging and validating parameter sets does not yield any improvements when compared to siloed training, there is no reason to participate in parameter markets. We provide theoretical and empirical analyses validating the improvements gained from participating in such markets.

How to monetize model parameters in a competitive market? In settings where parameters are bought and sold, rather than bartered, it can be challenging to price these assets. Both the seller and the buyer may not have a clear understanding of the other's valuation of the parameters, which makes it difficult for each to maximize their revenues in a trade. To address this issue, we apply a Bayesian-optimal pricing mechanism [23] to provide valuation of parameter sets and study Nash bargaining [24] to find market prices in negotiation.

Results and Contributions. We propose and formulate a viable marketplace to trade parameters for machine learning models. We validate it in both theoretical and empirical settings. Theoretically, in basic scenarios we show how agent training converges faster through purchasing parameters in the market. We offer bounds on the improvement gained via trading when training linear models. Empirically, we conduct experiments in a variety of practical scenarios to validate the framework's effectiveness. We demonstrate that compared to agents who stay outside the market and train models in isolation, participating in parameter trading, even trading subsets of the full set of model parameters, provides benefits to efficient model training and better model performance. For example, when training and trading parameters of ResNet20 on TinyImageNet, two agents improve their performance by gaining accuracy improvements of +10.02% and +15.93% versus separate training. We also demonstrate the success of price estimation to monetize parameters and negotiate prices.

2 Related Works

First, we describe two related concepts: data and model -as opposed to parameter -marketplaces. We then give background on model alignment techniques, which are used in our framework.

Data Marketplaces. Data is a key ingredient in machine learning pipelines. There is a rich vein of work proposing infrastructure to trade data as a commodity [21, 22, 25, 26, 27, 28, 29]. Such marketplaces have also been instantiated in industry, including in services such as Amazon Web Services (AWS) Data Exchange, Microsoft's Azure Data Marketplace, and Google's Cloud Data Catalog. Such research; however, cannot be directly applied to trading parameters. It is relatively easy to evaluate the valuation of a data trade using basic statistical measurements. In contrast, the value of parameters is challenging to measure, as it can only be determined after testing and depends on the model's performance.

Model Marketplaces. There have been several efforts to build markets to trade entire model instances trained by a centralized broker [30, 31]. Two major obstacles that need to be overcome are determining the value and pricing models, and safeguarding privacy. To address the former issue, a noise-injection mechanism has been suggested, which involves assessing the accuracy of an optimal model with random Gaussian noise to determine its worth and creating a price-error curve for selling it on the market [30]. The latter issue has been tackled by proposing a system that apply differential privacy while still maximizing revenue [31]. In contrast to trading entire model instances for downstream use, parameter markets are far more refined, enabling each user in the market to train their own models for their own purposes while gaining from others' training runs.

Model Alignment. Models that are trained with different batch orders or initialization weights may not be properly aligned. Directly merging purchased parameters through interpolation may fail. The process of aligning parameters in model training is therefore critical. Recent studies have explored the geometric relationship between models [11, 12, 13, 14, 15] and proposed methods for aligning two sets of neural network parameters [16, 17]. We use such techniques as a building block in our proposed market infrastructure. Through proper model alignments, we expect agents are able to find and purchase desired parameter sets in the market.

3 A Framework for Parameter Markets

We provide a general description of the proposed marketplace and then discuss each component in depth.

3.1 General Marketplace Framework

Figure 1 depicts a two-agent version of the marketplace. Multiple agents training models for potentially different tasks seek to buy or sell sets of parameters. Buying well-trained parameter sets enables reaching a goal performance faster while selling such parameters produces profits.

A high-level description of the trading process follows. First, agents send their own parameters to the market 1 . A third party (the broker) operates a "try-before-purchase" routine to align and merge parameters for buyers 2 . The broker privately informs agents of the gains or losses resulting from a potential trade using validation data 3 . Based on this information, a buyer values a seller's parameters, and then makes a trading decision. If the buyer is willing to purchase, a quote request is sent out, and both sides generate and report their valuations to the broker 4 , 5 . The broker helps both parties negotiate the price until they reach an agreement 6 . Afterwards, the broker ships parameters to the buyer and transfers the payment to the seller, completing the trade.

3.2 Market Setup

In the following sections, we discuss each foundational concept in parameter markets. We first fix some notation. For agent u , let D u = { s u,i } n u i =1 be the samples drawn from a distribution D u supported on S . At round t , let θ t u ∈ R d be trained parameters that agent u has access to, and let ˆ L u ( θ t u ) be the empirical loss of agent u for the corresponding model measured on data D u . The

Pricing Parameters

Pricing Parameters

$

$

AgentA

AgentB

O

Q

Send Gain-from-Trade

SendParameters

Broker

SendParameters

Send Quotation Request

Send Valuations

SendValuations

Try-before-purchase

Price Negotiation

empirical loss is defined with the agents' loss function /lscript in a standard fashion,

While our framework can handle general settings (i.e., any well-trained parameter sets can be traded), we focus on supervised learning for ease of exposition. We assume S = X × Y where X and Y are the instance and label spaces respectively. Thus, s u,i = ( x u,i , y u,i ) , and also denote D u as ( X u , Y u ) where X u is the set of points x u,i and Y u is the set of corresponding labels. We drop the superscript u when it is otherwise clear from the context.

Agents and Broker. For simplicity, suppose there are two agents A,B in the market. Agent A has n a data points ( X a , Y a ) , and agent B has n b data points ( X b , Y b ) . There is also a trusted third party, the broker Z , helping the two agents operate in the market impartially. The broker has access to a validation dataset ( X z , Y z ) of size n z with noiseless samples from both distributions.

In the proposed trading framework, the two agents train their models locally first and then trade with each other. At the beginning of round t , agents perform a standard gradient descent update using their own datasets. That is, ˙ θ t u := θ t -1 u -η ∇ ˆ L u ( θ t -1 u ) , u ∈ { a, b } , where η indicates the step size.

Using Purchased Parameters. In the market, parameters can be bought or sold. To use acquired parameters, several model alignment techniques can be employed [16, 17]. For a potential trade, broker aligns seller's parameter sets to match a buyer's. Post-alignment, the parameters can be merged as a straightforward convex combination. For instance, if agent A is willing to buy parameters from agent B , and a trade occurs eventually, the post-alignment combination will have the form ¯ θ t a := (1 -α ) ˙ θ t a + α ˙ θ t b , α ∈ (0 , 1] . Here α and (for agent B , β ) are weights indicating the portion of trained parameters from the seller side in the merged parameters.

3.3 Try-Before-Purchase Mechanism

However, before making any trading decision, agents are unaware of the potential benefits they can achieve through purchasing. Our proposal is for a neutral broker to implement a "try-before-purchase" mechanism which assists agents. It does so by helping them align and merge parameters then pre-evaluate their quality by using the broker's dataset ( X z , Y z ) . In the try-before-purchase mechanism, the broker can pick the optimized weights α or β for buyer agents by minimizing the empirical loss. We write α := arg min ν ∈ (0 , 1] ˆ L z ( (1 -ν ) ˙ θ t a + ν ˙ θ t b ) and β := arg min ν ∈ (0 , 1] ˆ L z ( (1 -ν ) ˙ θ t b + ν ˙ θ t a ) .

Using the optimized purchased weights, the broker calculates and communicates to agents their gain or loss from the trade in a confidential manner. We denote the gain-from-trade for agent u by ∆ t u , u ∈ { a, b } , which serves as prior knowledge to help agents make informed decisions about whether to make purchases or not. Generally, the notion of gain-from-trade is to compare relative improvement versus not trading. The trading benefits can be expressed in various ways, such as the difference between agent's loss before and after the trade. For example, it might take the form ∆ t u = ˆ L z ( ˙ θ t u ) -ˆ L z ( ¯ θ t u ) . Other ways to define the gain-from-trade include using the relative improvement ratio on the empirical loss or improvement ratio on the (estimated) parameter recovery error.

If the gain-from-trade ∆ t u does not indicate any benefit for buyer agent u , the agent will not be willing to make a purchase. Consequently, no quote request will be sent out, and no trade will take place. In such a scenario, the parameters ˙ θ t u will remain with agent u until the next round of gradient descent. The final parameters at the end of round t , denoted by θ t u , can be either ¯ θ t u or ˙ θ t u . To indicate a trade, we use the indicator variable I t u :

3.4 Valuation and Pricing Mechanism

Once the gain-from-trade has been determined and trading decisions have been made, quote requests with purchased weights are circulated throughout the market. Both buyers and sellers begin to assess various parameters in order to negotiate prices. Each agent has their own private valuation function, denoted by v u : R d ↦→ R + , which quantifies their trading benefits produced by specific parameters θ ∈ R d . The valuation function of each agent is private and is not exposed to others. For instance, agent A 's valuation function is denoted by v a ( ˙ θ t b ) for the value of agent B 's parameters for purchasing, and v a ( ˙ θ t a ) is the value of their own parameters for selling. In order to generate revenue from a trade, v a ( ˙ θ t b ) is the highest price that agent A is willing to bid for purchase, while v a ( ˙ θ t a ) is the lowest price that agent A is willing to ask for in order to sell.

Once valuations are completed, the broker-in their role as an impartial middleman-assists in bargaining the market price to maximize revenue for both sides. The negotiation process for prices continues until a mutually acceptable market price for parameters is reached. When a buyer offers to pay a lower price for certain parameters than the seller is willing to accept, the trade cannot be fulfilled. To analyze this negotiation process, we treat it as a Nash bargaining problem [24]. We assume that the broker is knowledgeable about the valuations of both parties (not the valuation functions themselves) and sets the price by maximizing the revenue of both agents impartially. An agent's revenue, defined as U u , is derived from two sources: the profit earned from selling self-parameters and the profit gained from buying parameters. We employ the popular Cobb-Douglas function to maximize both agents' revenue [32]. We denote the market price for agent u 's parameters at round t as P t u ∈ R + , then formulate the problem accordingly. Set U a ( P t a , P t b ) := ( P t a -v a ( ˙ θ t a ) ) + ( v a ( ˙ θ t b ) -P t b ) and U b ( P t b , P t a ) := ( P t b -v b ( ˙ θ t b ) ) + ( v b ( ˙ θ t a ) -P t a ) . Then we have the problem

By solving this problem, the broker can determine the difference in price, denoted by ∆ P t ab , using

The steps for solving this problem are shown in the Appendix A. The resulting price difference ∆ P t ab represents the amount of money that needs to be transferred between parties.

4 Instantiating the Market: Concrete Examples

Now we give a concrete instantiation of the market that we described in the previous section.

Valuations. There are various ways for agents to define valuation functions (i.e. based on agent's preference, training budget, or model performance). Here we assume that in the market, agents to purchase use gain-from-trade ∆ t u , which can be seen as a notion of relative performance improvement, to assess the value of parameters so that v u ( ˙ θ t u ′ ) = ∆ t u , where u ′ represents their seller agent.

However, assessing the value of self-parameters for agent u , who is a seller, is a difficult task as there is no clear information available from the broker regarding the quality of such parameters. The best approach for a seller to maximize profit is to set a price as close as possible to the buyer's valuation, which is the highest price that the buyer is willing to pay. To arrive at this virtual valuation , we use the Bayesian-optimal pricing mechanism described in Myerson's work [23]. This mechanism enables the seller to monetize self-parameters. Under this Bayesian mechanism, we assume that the seller is also aware that the buyer's valuation arises from gain-from-trade, and that their valuation function is derived from a known probability distribution. We discuss these common priors in Appendix G.

Suppose the buyer's valuation has a cumulative distribution function F v . If the seller sets a price of P , the probability that the buyer is willing to purchase is P ( P < v ) = 1 -P ( v ≤ P ) = 1 -F v ( P ) . The expected revenue for the seller is P × (1 -F v ( P )) . Hence, the optimal price to ask for can be

found by maximizing the expected revenue, and it satisfies

Linear Model Case Study. To further illustrate a concrete example of the seller's virtual valuation, we consider pricing parameters for training linear models. For simplicity, we assume that the broker knows the true parameters θ ∗ -though this is not necessary in practice, as these can be approximated using the broker's validation dataset. Both agents' data is sampled from the same distribution so that true parameters are identical. Additionally, we take the gain-from-trade revealed by broker to be the ratio of squared parameter estimation error. We write

If ∆ t u > 1 , the agent's model is able to get closer to the true parameter ( θ ∗ ) by purchasing. We use agent A to illustrate the sale of their parameters ˙ θ t a to agent B . First, we show how agent A determines bounds for the buyer's valuation v b ( ˙ θ t a ) . Recall that once agent B expresses a willingness to purchase, a quote request with their purchased weight β will be communicated to agent A through the broker. Therefore, when agent A evaluates self-parameters, the broker provides three pieces of information: the buyer's purchased weight β , agent A 's purchased weight α , and the gain-from-trade ∆ t a . In this setting, we have that

Theorem 4.1. Bounds on Buyer's Gain-from-Trade : In the linear model setting, by knowing ∆ t a and weights α , β , agent A can obtain bounds on the gain-from-trade of agent B given by:

Discussion. Theorem 4.1 states that by knowing information disclosed by the broker (including purchased weights α , β ), the seller agent A can find the upper and lower bounds of the buyer's gain-from-trade, ∆ t b . This information can then be used to estimate the value that the buyer places on the item. Using the Bayesian optimal-pricing mechanism, and taking into account the known probability distribution, the seller can estimate the price to determine their own virtual valuation.

Furthermore, the optimal scenario occurs when the seller agent values self-parameters exactly as the buyer does. In this case, based on Eq. (1), the broker will set the transfer payment as ∆ P t ab = P t a -P t b = v b ( ˙ θ t a ) -v a ( ˙ θ t b ) . Hence, the transfer payment is equal to the difference between the gain-from-trade of the two agents, where ∆ P t ab = ∆ t a -∆ t b .

The proof for Theorem 4.1 is in the Appendix C. We summarize trading steps in a single round for this instantiation above (Algorithm 1), where two agents are training and trading parameters for the linear model setting. Here, agents A,B act as a seller and a buyer, respectively.

5 Convergence Analysis

Next, we study a theoretical analysis for the effectiveness of buying parameters . In particular, we are interested in understanding whether participating in the market leads to faster training convergence (in the worst-case scenario). We show this holds in a simplified setting where agent A always leads and never purchases parameters, while agent B always purchases from agent A . This asymmetry could be due to various reasons including a lack of training resources for agent B . Here we study a setting for general L-smooth functions, which is more practical, as the broker doesn't need to be knowledgeable about the true parameter θ ∗ . We assume the broker's loss is lower than agents' losses, in particular, ˆ L z ( θ ) ≤ ˆ L a ( θ ) and ˆ L z ( θ ) ≤ ˆ L b ( θ ) , ∀ θ ∈ R d . We take the gain-from-trade ∆ t u by using the subtraction of empirical loss before and after a trade. We write it as

Theorem 5.1. For all agents u ∈ { a, b, z } , let the loss function ˆ L u be L -smooth and let the samples on all agents be drawn from the same distribution D . Let E D [ ˆ L u ] = L u , and ∆ t b = ˆ L z ( ˙ θ t b ) -ˆ L z ( ¯ θ t b ) . Let the algorithm run until round T with step size η ∈ (0 , 1 L ) , and let δ b := min t ∈ [ T ] E [ ∆ t b ] and ¯ g 2 b := min t ∈ [ T ] E [ ‖∇ ˆ L b ( θ t b ) ‖ 2 2 ] . Then we have the following results,

(Always Trade) If ∆ t b > 0 , ∀ t , and agent B always buys (i.e. I t b = 1 , ∀ t ). Then T ≥ 2( L ( θ 0 b ) -L ( θ ∗ b )) η/epsilon1 2 +2 δ b implies ¯ g b ≤ /epsilon1 .

(Never Trade) If the agents never trade i.e. ( I t a = I t b = 0 , ∀ t ). Then ¯ g b ≤ /epsilon1 for T ≥ 2( L ( θ 0 b ) -L ( θ ∗ b )) η/epsilon1 2 .

Discussion. We show the convergence rate of always trade for agent B is O ( 1 / ( /epsilon1 2 + δ b ) ) , while never trade is O (1 / /epsilon1 2 ) . The difference between these two scenarios is due to δ b -the minimal gain-from-trade over T runs. More gain-from-trade implies a smaller T (i.e. faster convergence) in the worst-case scenario. In addition, when δ b is Ω ( /epsilon1 ) , we can get much better convergence rate of O (1 / /epsilon1 ) . Our results in this fundamental setting illustrate that participation in the market can lead to faster convergence when there exists gain-from-trade. The proof for Theorem 5.1 is in the Appendix D.1. In addition to this general setting, we provide convergence analysis for the linear model that we used as a case study in Sec. 4. This analysis also shows a better convergence rate when agents trade parameters. See Appendix D.2 for more details.

6 Experiments

We study the proposed framework empirically. Our goals are to validate (i) trading in the proposed framework results in improvements, (ii) these improvements persist even when trading subsets of parameters, (iii) these persist even when agents are trading different models for different tasks, (iv) trading and pricing are viable in competitive settings, and (v) understand the importance of key components of our framework, such as the need for alignment.

6.1 Collaborative Agents

We first conduct experiments in a collaborative setting, where there is no payment for buying parameters. Our goal is to validate the potential improvements from transactions independently of the pricing mechanism, which we explore in a later set of experiments.

6.1.1 Parameter Trading in Neural Networks

Setup. We use MNIST [33], CIFAR10 [34], and TinyImageNet [35] for training MLPs and ResNet20 [36]. Agents have imbalanced datasets where half of the classes contain only 10% of datapoints. Agents are limited to collecting a part of a dataset, making it difficult for them to achieve satisfactory performance without collaborating and trading parameters to leverage each other's strengths.

Models are trained from different random initializations and batch orders over 60 epochs. Agents trade entire parameter sets and join the market after five epochs. The broker discloses gain-from-trade to agents. Broker aligns parameters [16], then merge.

In addition to the approach that agents train models on their own, we include another baseline method, FedAvg [37], which assumes that there is no broker involved in a trade to help agents align parameters and optimize their purchased weights. In FedAvg, the interpolated weight is determined by the portion of data assets that an agent is endowed with, which is 0.5 in this setting.

Results. Table 1 shows the performance of two agents. We find that both agents are able to achieve improved performance by leveraging each other's training expertise, compared to out-of-market agents. Specifically, training and trading ResNet20 with TinyImageNet resulted in improving accuracy by +10.02% and +15.93% , respectively. We measure two ways to merge parameters for buying: with and without model alignment. With model alignment, the broker is able to merge models more effectively, ultimately enhancing performance for both agents. In addition, compared to FedAvg method, results confirm the significance of having a trusted broker in parameter trading. Without an intermediary broker to facilitate the trade, the performance of purchased weights can be negatively impacted, as evidenced by the results of CIFAR10 + ResNet20 and TinyImageNet + ResNet20.

Finally, Figure 2 displays a comparison of testing loss for the MLP on MNIST. Our results demonstrate that trading parameters results in faster convergence compared to siloed training. Using alignment further helps, as expected.

6.1.2 Parameter Subsets Trading in Neural Networks

Setup. Next, we explore the potential benefits of trading subsets of parameters. This scenario may take place when agents are interested in only certain components or are restricted by trading budgets. We use the same data endowment as in the preceding configuration. We train two 5-layer MLPs on MNIST and align parameters in each layer to trade.

Results. Table 2 displays the results by trading parameters from different layers. As expected, trading the entire model gives optimal performance (the last row). Trading subsets is helpful but suboptimal. We observe that purchasing parameters from shallow layers, close to the input layers,

offers more benefits than trading with deeper layers. This provides trading guidance for budget-limited agents.

6.1.3 The Effectiveness of Buying Parameters in Controlled Settings

Setup. We use a synthetic dataset to study trading in a fully-controlled environment. We use two linear regression models with dimension d = 1000 . Agent A has n a = 500 datapoints in dataset ( X a , Y a ) and B has n b = 800 datapoints in dataset ( X b , Y b ) , but the latter's labels are affected by zero-mean Gaussian noise ( σ 2 = 0 . 5 ). We assume that the broker knows the true parameter θ ∗ and obtains ( X z , Y z ) and has access to n z = 10 , 000 datapoints. Both agents start learning function f a , f b with the same initialized weight θ 0 . We compare the results over 100 runs with agents who obtain the same data endowment but do not participate in the market.

Results. The leftmost Figure 3 displays the trading log over 100 runs. Green dots and red dots show whether an agent purchases the other's parameters in a specific run. Next, we show the ratio of squared parameter estimation error between agents. If the ratio is larger than 1 (red dashed line), agent B leads the market. We see that agent A leads the market in the first half of runs, making agent B continue buying parameters from agent A . At the end of a few runs, agent B turns to the lead.

The rightmost plots show convergence. If an agent is involved in the market and trades with the other, the convergence rate is faster compared to never trading. This study demonstrates agents' behaviors in the trading runs and validates the effectiveness of buying parameters, leading to more efficient model training. We compute the empirical testing loss at the end. Compared to out-of-market agents, we find that agent A and agent B are able to improve testing loss by 42.84% and 23.88% , respectively.

6.1.4 Trading Parameters of Models with Different Purposes

Setup. Next, we validate whether trading makes sense even if agents are training different models for different purposes . This scenario is more realistic as it simulates situations where organizations in the market are working on different but potentially related tasks. We model this setting by sweeping the distance between the true parameters θ ∗ a , θ ∗ b of two linear regression models to observe how it impacts the benefits of trading.

Results. We record the benefits from trading when compared to an agent who does not participate but has the same data as agent

A . We measure the relative improvement in empirical testing loss. The results are shown in Figure 4, which indicates that even though the two agents are not training on the same task, agent A is still able to benefit from trading. Note that the gain exists even when the tasks are quite different (i.e. large distance ‖ θ ∗ a -θ ∗ b ‖ 2 ) but is strongest as the tasks are most closely related.

6.2 Competitive Agents

Finally, we study agents engaging in a competitive scenario. In this case, the transactions require pricing. We validate the proposed pricing mechanism.

6.2.1 The Effectiveness of Bayesian Optimal Pricing Mechanism

Setup. We reuse the synthesized dataset from the collaborative experiment. We set the buyer's valuation to gain-from-trade and estimate the seller's virtual valuation by the lower bound that we can find.

Results. Market prices negotiated by the broker over 100 trading runs are displayed in Figure 5. Based on Eq. 1, the market price is determined by the average of the buyer's valuation and the seller's virtual valuation. To demonstrate market efficiency, we also study the scenario where the seller sets the valuation to be exactly the same as the buyer's. As shown in Figure 3 (second from left: performance ratio), agent A is initially leading in the first half of the runs, resulting in a higher price for their parameters. However, at the end of a few runs, agent B takes the lead, causing their parameter price to increase while opponent A ' goes down. It is important to note that the gap between the estimated price (the blue line) and the optimal price (the orange line) can be reduced with more information learned through the market, such as via historical transactions. Besides, the resulting negotiated price creates a discrepancy with the price in the optimal scenario where both parties report their valuations truthfully, highlighting the significance of revealing accurate parameter values and justifying the need for incentives. At last, this study illustrates the feasibility of using the Bayesian optimal pricing mechanism to assist seller agents in monetizing parameters with limited information to make a trade.

7 Conclusion

In this paper, we introduced a framework for parameter markets that can serve to reduce the heavy costs of large-scale model training. Borrowing from economic principles, we provided a set of mechanisms that enable the functioning of such markets. Theoretically, for simple settings, we analyzed market efficiency and proved that agents can gain from participating. We empirically validated the effectiveness of parameter markets under collaborative and competitive settings and demonstrated when participants in the market earn mutual benefits through market usage.

8 Acknowledgments

We would like to express our gratitude to the Wisconsin Alumni Research Foundation (WARF) for supporting this work. Additionally, we would like to thank all the reviewers for their valuable comments and constructive feedback on our work.

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21(140):1-67, 2020.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 , 2022.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

Or Sharir, Barak Peleg, and Yoav Shoham. The cost of training nlp models: A concise overview. arXiv preprint arXiv:2004.08900 , 2020.

Lie He, An Bian, and Martin Jaggi. Cola: Decentralized linear learning. Advances in Neural Information Processing Systems , 31, 2018.

Hanlin Tang, Xiangru Lian, Ming Yan, Ce Zhang, and Ji Liu. d 2 : Decentralized training over decentralized data. In International Conference on Machine Learning , pages 4848-4856. PMLR, 2018.

Qinyi Luo, Jinkun Lin, Youwei Zhuo, and Xuehai Qian. Hop: Heterogeneity-aware decentralized training. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems , pages 893-907, 2019.

Binhang Yuan, Yongjun He, Jared Quincy Davis, Tianyi Zhang, Tri Dao, Beidi Chen, Percy Liang, Christopher Re, and Ce Zhang. Decentralized training of foundation models in heterogeneous environments. arXiv preprint arXiv:2206.01288 , 2022.

Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems , 31, 2018.

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning , pages 3259-3269. PMLR, 2020.

Norman Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Optimizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems , 33:15300-15311, 2020.

Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. arXiv preprint arXiv:2010.04495 , 2020.

Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06296 , 2021.

Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.04836 , 2022.

Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. Repair: Renormalizing permuted activations for interpolation repair. arXiv preprint arXiv:2211.08403 , 2022.

Min Du, Ruoxi Jia, and Dawn Song. Robust anomaly detection and backdoor attack detection via differential privacy. arXiv preprint arXiv:1911.07116 , 2019.

Xudong Zhu and Hui Li. Privacy-preserving decentralized federated deep learning. In ACM Turing Award Celebration Conference-China (ACM TURC 2021) , pages 33-38, 2021.

Dario Pasquini, Mathilde Raynal, and Carmela Troncoso. On the privacy of decentralized machine learning. arXiv preprint arXiv:2205.08443 , 2022.

Qiyang Song, Jiahao Cao, Kun Sun, Qi Li, and Ke Xu. Try before you buy: Privacy-preserving data evaluation on cloud-based machine learning data marketplace. In Annual Computer Security Applications Conference , pages 260-272, 2021.

Santiago Andrés Azcoitia and Nikolaos Laoutaris. Try before you buy: A practical data purchasing algorithm for real-world data marketplaces. In Proceedings of the 1st International Workshop on Data Economy , pages 27-33, 2022.

Roger B Myerson. Optimal auction design. Mathematics of operations research , 6(1):58-73, 1981.

John F Nash Jr. The bargaining problem. Econometrica: Journal of the econometric society , pages 155-162, 1950.

Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning , pages 2242-2251. PMLR, 2019.

Anish Agarwal, Munther Dahleh, and Tuhin Sarkar. A marketplace for data: An algorithmic solution. In Proceedings of the 2019 ACM Conference on Economics and Computation , pages 701-726, 2019.

Raul Castro Fernandez, Pranav Subramaniam, and Michael J Franklin. Data market platforms: Trading data assets to solve data problems. arXiv preprint arXiv:2002.01047 , 2020.

Junjie Chen, Minming Li, and Haifeng Xu. Selling data to a machine learner: Pricing via costly signaling. In International Conference on Machine Learning , pages 3336-3359. PMLR, 2022.

Meena Jagadeesan, Michael I Jordan, and Nika Haghtalab. Competition, alignment, and equilibria in digital marketplaces. arXiv preprint arXiv:2208.14423 , 2022.

Lingjiao Chen, Paraschos Koutris, and Arun Kumar. Towards model-based pricing for machine learning in a data marketplace. In Proceedings of the 2019 International Conference on Management of Data , pages 1535-1552, 2019.

Jinfei Liu, Jian Lou, Junxu Liu, Li Xiong, Jian Pei, and Jimeng Sun. Dealer: an end-to-end model marketplace with differential privacy. 2021.

Charles W Cobb and Paul H Douglas. A theory of production. 1928.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

ABDULLAH YAVA¸ S. Economics of brokerage: an overview. Journal of Real Estate Literature , 2(2):169-195, 1994.

Simran Khanuja, Melvin Johnson, and Partha Talukdar. Mergedistill: Merging pre-trained language models using distillation. arXiv preprint arXiv:2106.02834 , 2021.

Sebastian Lawrenz, Priyanka Sharma, and Andreas Rausch. Blockchain technology as an approach for data marketplaces. In Proceedings of the 2019 international conference on blockchain technology , pages 55-59, 2019.

Franziska Boenisch. A systematic review on model watermarking for neural networks. Frontiers in big Data , 4:729663, 2021.

Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1-15. IEEE, 2022.

Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample-efficient model evaluation. In International Conference on Machine Learning , pages 5753-5763. PMLR, 2021.

Ante ´ Custi´ c, Vladyslav Sokol, Abraham P Punnen, and Binay Bhattacharya. The bilinear assignment problem: complexity and polynomially solvable special cases. Mathematical programming , 166:185-205, 2017.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel concept called *parameter markets*, which serves as a platform for exchanging parameters learned by machine learning models. In this framework, agents have the option to engage in parameter trading, to achieve (1) mutual benefits through collaboration or (2) monetary gains by simply selling these parameters. The key contributions of this paper involve the formulation and design of these parameter markets. This encompasses defining the value of parameters, establishing pricing mechanisms, and outlining how transactions can occur. The authors explore both competitive and collaborative scenarios and illustrate the advantages agents can obtain by opting in to trade through the marketplace.\n\nSTRENGTHS:\n1. The idea to trade parameters is novel (to my best knowledge) and is quite different from data markets. The key benefit from parameter markets is that agents can train models for their own tasks and can yet benefit from training runs of each other. This is a very relevant problem to study.\n2. The formulation is thorough and the empirical experiments are adequate to demonstrate the claims made in the paper\n3. The paper is well written and easy to follow.\n\nWEAKNESSES:\n1. This work is a bit existential - this framework works if *model privacy and alignment are assured*. \n2. Theorem 4.1 relies on the broker having access to $\\theta^*$ which is a strong assumption.\n3. There is no discussion relating to incentives (i.e. agent misreporting their valuations to benefit from the trades).Perhaps this paper could benefit from adding a mechanism design angle to this framework.\n\nQUESTIONS:\n1. Refer to Weaknesses. \n2. It's unclear why Cobb-Douglas utility function is used.\n\nLIMITATIONS:\nTo my best knowledge, there isn't an immediate potential negative societal impact of their work. This paper, however, doesn't have an explicit weakness section (relating to their framework).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n* This paper proposes an economic framework for trading parameters of prediction models.\n* Interaction is modeled as a brokered marketplace - Each agent $u$ trains a model characterized by parameters $\\theta_u$ using gradient descent. At each time-step $t$: \n  * Each agent performs a gradient descent step on the previous parameters $\\theta_u^{t-1}$, to obtain $\\dot{\\theta}_u^t$.\n  * After the GD step, agents relay their model parameters $\\dot{\\theta}_u^t$ to a  trusted broker, which calculates the optimal linear interpolation between them, yielding the possibly-improved combination $\\bar{\\theta}_u$ for each agent. The broker indicates for each agent the gain-from-trade $\\Delta_u^t$.\n  * The agents participate in Nash bargaining to decide on the value exchange, set trading prices, and $\\theta^t_u$ is decided. Each agent has a valuation function $v_u$ for their own parameters, and the other agent’s parameters. Utility function is assumed to be fully known by the broker. \n* Analysis:\n  * Analysis is restricted to the two-agent setting $u\\in\\{A,B\\}$, and a specific form for gain-from-trade $\\Delta_u^t$ is assumed. It is assumed that each seller has complete knowledge of the valuation prior, such that Myerson’s revenue-maximizing pricing mechanism (virtual valuations) can be applied based on the information relayed from the broker ($\\alpha$, $\\beta$, $\\Delta_a^t$ - By Theorem 4.1).\n  * It is assumed that the broker has complete knowledge of the optimal model parameters in advance ($\\theta^*$). Combining with the assumptions about the agents’ utility function, a closed-form expression is obtained for the monetary transfer after Nash bargaining.\n  * In Section 4.1, two upper bounds on convergence rates are presented - One for a scenario where agent $B$ always trades (Theorem 4.2), and one for a scenario where both agents don’t trade (Proposition 4.3). The “always-trade” bound (Thm 4.2) is lower, and the authors claim that this explains why trading is sometimes beneficial. \n* Two sets of experiments are presented:\n  * The first set of experiments is an evaluation of the data-sharing setting, without taking parameters into account (Section 5.1.1 - complete networks, Section 5.1.2 - subsets of networks), on real-world vision datasets (MNIST, CIFAR10, TinyImageNet).\n  * Finally, the value of trading is evaluated using a synthetic model, and positive results are reported for both cooperative and competitive scenarios.\n\nSTRENGTHS:\n* Problem is well-motivated.\n* Leverages recent deep insights about the structure of contemporary learning problems.\n* The breadth of work is substantial, and it contains a number of new definitions, theoretical analyses, and empirical evaluations.\n* Code is provided.\n\nWEAKNESSES:\n* Some key assumptions are not well supported:\n  * Valuation functions $v(\\theta)$ are assumed to be fully known by the broker. For contemporary neural nets, the dimensionality of $\\theta$ could be in the order of billions (or even trillions, such as in GPT4), and therefore the parameter valuation would be a function $v:\\mathbb{R}^{10^9}\\to\\mathbb{R}$ - Which may be prohibitively expensive to represent, store and compute.\n  * Broker is assumed to know the true model parameters $\\theta^*$ in advance, which in practice may render the whole learning process unnecessary. In L214, it is claimed that knowing $\\theta^*$ “is not necessary in practice”, but I did not find further support of this claim. \n  * Very specific functional forms are assumed for gain-from-trade but are not sufficiently justified.\n  * A common prior on agent valuation is assumed (L208), however it is not discussed whether having such prior is realistic and how it would be implemented.\n* Mathematical soundness concern regarding one of the claims: In section 4.1, authors claim that trading is beneficial by comparing two performance upper bounds (Theorem 4.2, Proposition 4.3). However, the difference between two upper bounds does not indicate the relation between the true quantities unless they are tight (in other words, proving that $a<100$ and $b<50$ does not indicate that $a>b$ unless there are matching lower bounds for $a$ and $b$). Therefore, if this observation is true, the analysis in Section 4.1 does not address the posed question (whether participation leads to better convergence).\n* First set of experiments (Sec 5.1.1, 5.12) only seem to demonstrate the ability to interpolate model parameters, but do contain simulation of parameter trading. Economic behavior is only demonstrated on the synthetic dataset. As the benefit of parameter interpolation has already been demonstrated in the literature to some extent, I did not understand the contribution of the results in sections 5.1.1, 5.1.2 to the understanding of parameter-trading markets.\n* Paper only analyzes the case of two agents $\\{A,B\\}$. Unclear how results extend to multiple agents.\n* Trading parameters during the gradient descent process would require agents to train models simultaneously, or suffer significant delays due to trading. This may make such trading impractical.\n\nQUESTIONS:\n* How would the model extend to more than 2 agents? Which results can be used as-is, and which need to be generalized? Would the system behave qualitatively differently in any way when the number of agents is increased?\n* What are the computational requirements from the broker?\n* Is it possible to quantify the economic implications of the broker not being able to interpolate successfully?\n\nLIMITATIONS:\nI feel that limitations could be discussed at more depth. The key assumptions are added gradually throughout the paper, and it is hard to keep track. I feel that the paper can greatly benefit from a thorough discussion of limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates how to design a marketplace for model parameters. The marketplace consists of agents training models for potentially different objectives and a trusted third-party broker. The broker receives the model parameters from each agent, assesses (and informs each agent of) the loss achieved by the merged model, and helps determine prices using Nash bargaining. \n\nThe paper provides a theoretical and empirical analysis of this framework. In particular, they prove a bound on the gain from trade for linear models and they analyze the effectiveness of buying parameters in terms of improving training convergence.  They empirically assess their framework on an image classification task. They validate that the merging method of the broker (Ainsworth et al., 2022) improves loss, and they evaluate the effectiveness of their pricing strategy.\n\nSTRENGTHS:\nThe idea of trading subsets of parameters in a marketplace seems novel and interesting. Parameter marketplaces seem to contrast with classical approaches of data marketplaces, where agents buy data from each other, or model marketplaces, where agents directly purchase models or model access.\n\nWEAKNESSES:\nThe proposed framework seems to rely on there being a trusted broker facilitating the trades and assessing and reporting the gains of merging parameters to the agents. In reality, this seems to be a strong assumption, as it is not clear if (a) a broker can be trusted with all of the parameters, and (b) if a broker will be able to assess the gain of merging models without significant data and infrastructure. See question below.  \n\nThe technical contribution of the paper seems limited. The paper combines existing work on merging and aligning models (e.g. Ainsworth et al., 2022) and classical approaches for Nash bargaining. However, the paper seems to use these approaches out-of-box and does not seem to present significant new technical contributions. \n\nThe paper is a bit difficult to follow and the exposition could be improved. For example, the results in Section 4 use a lot of notation and the qualitative insights are a bit difficult to infer.\n\nQUESTIONS:\nHow might the framework be modified to avoid the reliance on a trusted broker, e.g., via a more decentralized approach?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper proposes a framework for collaborative and competitive parameter trading among deep learning agents. The authors conduct experiments to validate the effectiveness of the proposed framework in improving the performance of the agents. The experiments show that even when the agents are training on different tasks, they can still benefit from trading parameters. The authors also validate the proposed pricing mechanism in a competitive scenario. The paper makes contributions in demonstrating the potential of parameter trading to improve the performance of deep learning agents and providing a framework for collaborative and competitive trading.

STRENGTHS:
- The paper proposes a novel framework for collaborative and competitive parameter trading
- The experimental results are promising.
- The experiments show that even when the agents are training on different tasks, they can still benefit from trading parameters.

WEAKNESSES:
- The authors do not provide an analysis of the limitations of the proposed framework, which could help identify potential issues in real-world scenarios.
- The experiments could benefit from a more extensive evaluation of the proposed framework's performance by comparing it with more baselines.
- The paper could benefit from a more detailed explanation of the proposed pricing mechanism and its implementation.
- The authors do not discuss the potential ethical implications of parameter trading, which could be relevant in real-world applications.

QUESTIONS:
- Why is parameter trading necessary?
- How to resolve the legal concerns on copyrights?

LIMITATIONS:
More limitations should be discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10