SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition 3.1. The closed-form solution for the inner-maximization in (12) for each  $x$  is given by
$$
\hat {w} _ {\nu_ {i}} ^ {*} (x) = \exp \left(\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} (x) - 1\right) \tag {13}
$$
By plugging equation (13) into (12), we obtain the following minimization problem:
$$
\min  _ {\nu_ {i}} \bar {\rho} _ {i} \alpha \hat {\mathbf {E}} _ {x \in D _ {\rho_ {i}}} [ \exp (\frac {1}{\alpha} \hat {e} _ {\nu_ {i}} (x) - 1) ] + (1 - \gamma) \mathbb {E} _ {s _ {0} \sim p _ {0}} [ \nu_ {i} (s _ {0}) ] =: L (\nu_ {i}). \tag {14}
$$
As we show in Proposition B.1 in the Appendix,  $\tilde{L}(\nu_i)$  is an unconstrained convex optimization problem where the function to learn  $\nu_i$  is state-dependent. Furthermore, the terms in (14) are estimated only using the  $(s, a_i, \mathbf{a}_{-i}, s')$  samples in the dataset, making it free from the extrapolation error by bootstrapping OOD action values. Also, since  $\nu_i(s)$  does not involve joint actions, it is not required to adopt IGM-principle in  $\nu_i$  network modeling; thus, there is no need to limit the expressiveness power of the function approximator. In practice, we parameterize  $\nu_i$  using simple MLPs, which take the state  $s$  as an input and output a scalar value.
Policy Extraction The final remaining step is to extract a policy from the estimated distribution correction ratio  $w_{i}^{*}(s,a_{i}) = \frac{d^{\pi_{i}^{*}}(s,a_{i})}{d^{D}(s,a_{i})}$ . Unlike actor-critic approaches which perform intertwined optimizations by alternating between policy evaluation and policy improvement, solving (14) directly results in the optimal  $\nu_{i}^{*}$ . However, this does not result in an executable policy. We therefore utilize the I-projection policy extraction method from [14] which we found to be most numerically stable
$$
\begin{array}{l} \underset {\pi_ {i}} {\arg \min } \mathrm {D} _ {\mathrm {K L}} \left(d ^ {D} (s) \pi_ {i} \left(a _ {i} \mid s\right) \boldsymbol {\pi} _ {- i} (\mathbf {a} _ {- i} \mid s) \| d ^ {D} (s) \pi_ {i} ^ {*} \left(a _ {i} \mid s\right) \boldsymbol {\pi} _ {- i} (\mathbf {a} _ {- i} \mid s)\right) (15) \\ = \underset {\pi_ {i}} {\arg \min } \hat {\mathbf {E}} _ {s \in D, a _ {i} \sim \pi_ {i}} \left[ - \log w _ {i} ^ {*} (s, a _ {i}) + \mathrm {D} _ {\mathrm {K L}} \left(\pi_ {i} \left(a _ {i} | s\right) \mid \mid \pi_ {i} ^ {D} \left(a _ {i} | s\right)\right) \right] (16) \\ \end{array}
$$
In summary, AlberDICE computes the best response policy of agent  $i$  by: (1) resampling data points based on the other agents' policy ratios  $\rho$  (11) where the data policy  $\pi_{-i}^{D}(\mathbf{a}_{-i}|s,a_{i})$  can be pretrained, (2) solving a minimization problem to find  $\nu_{i}^{*}(s)$  (31) and finally, (3) extracting the policy using the obtained  $\nu_{i}^{*}$  by I-projection (15). In practice, rather than training  $\nu_{i}$  until convergence at each iteration, we perform a single gradient update for each agent  $\nu_{i}$  and  $\pi_{i}$  alternatively. We outline the details of policy extraction (Appendix E.2) and the full learning procedure in Algorithm 1 (Appendix E).
4 Preservation of Common Rewards and Convergence to Nash Policies
In the previous sections, AlberDICE was derived as a practical algorithm in which agents alternately compute the best response DICE while avoiding OOD joint actions. We now prove formally that this procedure converges to Nash policies. While it is known that alternating best response can converge to Nash policies in common reward settings [1], it is not immediately clear whether the same result holds for the regularized LP (2-3), and hence the regularized reward function of the environment, preserves the common reward structure of the original MMDP. As we show in Lemma 4.1, this is indeed the case, i.e., the modified reward in (2-3) is shared across all agents. This directly implies that optimization of the corresponding LP yields the same value for all agents  $i \in \mathcal{N}$  for any joint policy,  $\pi$ , with factorized individual policies,  $\{\pi_i\}_{i \in \mathcal{N}}$ .
Lemma 4.1. Consider a joint policy  $\pi = (\pi_i)_{i\in \mathcal{N}}$ , with factorized individual policies, i.e.,  $\pi (\mathbf{a}|s) = \prod_{i\in \mathcal{N}}\pi_i(a_i|s)$  for all  $(s,\mathbf{a})\in S\times \mathcal{A}$  with  $\mathbf{a} = (a_i)_{i\in N}$ . Then, the regularized objective in the LP formulation of AlberDICE, cf. equation (2), can be evaluated to
$$
\sum_ {s, a _ {i}, \mathbf {a} _ {- i}} d _ {i} ^ {\pi} (s, a _ {i}) \pi_ {- i} (\mathbf {a} _ {- i} | s) \tilde {r} (s, a _ {i}, \mathbf {a} _ {- i}),
$$
with  $\tilde{r}(s, a_i, \mathbf{a}_{-i}) := r(s, a_i, \mathbf{a}_{-i}) - \alpha \cdot \log \frac{d^\pi(s) \pi(\mathbf{a}|s)}{d^D(s, a_i, \mathbf{a}_{-i})}$ , for all  $(s, \mathbf{a}) \in S \times \mathcal{A}$ . In particular, for any joint policy,  $\pi = (\pi)_{i \in \mathcal{N}}$ , with factorized individual policies, the regularized objective in the LP formulation of AlberDICE attains the same value for all agents  $i \in \mathcal{N}$ .
We can now use Lemma 4.1 to show that AlberDICE enjoys desirable convergence guarantees in tabular domains in which the policies,  $\pi_i(a_i|s)$ , can be directly extracted from  $d_i(s,a_i)$  through the expression  $\pi_i(a_i|s) = \frac{d_i(s,a_i)}{\sum_{a_j} d_i(s,a_j)}$ .
Theorem 4.2. Given an MMDP,  $G$ , and a regularization parameter  $\alpha \geq 0$ , consider the modified MMDP  $\tilde{G}$  with rewards  $\tilde{r}$  as defined in Lemma 4.1 and assume that each agent alternately solves the regularized LP defined in equations (2-3). Then, the sequence of policy updates,  $(\pi^t)_{t \geq 0}$ , converges to a Nash policy,  $\pi^* = (\pi_i^*)_{i \in \mathcal{N}}$ , of  $\tilde{G}$ .
The proofs of Lemma 4.1 and Theorem 4.2 are given in Appendix D. Intuitively, Theorem 4.2 relies on the fact that the objectives in the alternating optimization problems (2-3) involve the same rewards for all agents for any value of the regularization parameter,  $\alpha \geq 0$ , cf. Lemma 4.1. Accordingly, every update by any agent improves this common value function,  $(\tilde{V}^{\pi}(s))_{s\in S}$ , and at some point the sequence of updates is bound to terminate at a (local) maximum of  $\tilde{V}$ . At this point, no agent can improve by deviating to another policy which implies that the corresponding joint policy is a Nash policy of the underlying (modified) MMDP. For practical purposes, it is also relevant to note that the process may terminate at an  $\epsilon$ -Nash policy (cf. Definition 2.1), since the improvements in the common value function may become arbitrarily small when solving the LPs numerically.
Table 1: Policy values after convergence for the Matrix Game in Figure 2 for  $D = \{ AA,AB,BA\}$
A direct implication from the construction of the AlberDICE algorithm, which is also utilized in the proof of Theorem 4.2, is that the AlberDICE algorithm maintains during its execution and, thus, also returns upon its termination, a factorizable policy, i.e., a policy that can be factorized.
Corollary 4.3. Let  $\pi^*$  be the extracted joint policy that is returned from the AlberDICE algorithm. Then,  $\pi^*$  is factorizable, i.e., there exist individual policies,  $\langle \pi_i^*\rangle_{i\in \mathcal{N}}$ , one for each agent, so that  $\pi^*(\mathbf{a}|s) = \prod_{i\in \mathcal{N}}\pi_i^*(a_i|s)$  for all  $\mathbf{a} \in A$ ,  $s \in S$ .
We evaluate AlberDICE on a series of benchmarks, namely the Penalty XOR Game and Bridge [5], as well as challenging high-dimensional domains such as Multi-Robot Warehouse (RWARE) [27], Google Research Football (GRF) [12] and StarCraft Multi-Agent Challenge (SMAC) [31]. Our baselines include Behavioral Cloning (BC), ICQ [40], OMAR [25], MADTKD [35] and OptiDICE [14]  $^{7}$ . For a fair comparison, all baseline algorithms use separate network parameters  $^{8}$  for each agent and the same policy structure. Further details on the dataset are provided in Appendix F.
5.1 Penalty XOR Game
We first evaluate AlberDICE on a  $2 \times 2$  Matrix Game called Penalty XOR shown in Figure 2. We construct four different datasets: (a)  $\{AB\}$ , (b)  $\{AB, BA\}$ , (c)  $\{AA, AB, BA\}$ , (d)  $\{AA, AB, BA, BB\}$ . The full results showing the final joint policy values are shown in Table 6 in the Appendix. We show the results for dataset (c) in Table 1.
AlberDICE is the only algorithm that converges to a deterministic optimal policy,  $BA$  or  $AB$  for all datasets.
On the other hand, OptiDICE and MATLAB converge to a stochastic policy where both agents choose  $A$  and  $B$  with equal probability. This is expected for both algorithms which optimize over joint actions during
Figure 2: Penalty XOR
centralized training which can still lead to joint action OOD if the joint policy is not factorizable. ICQ converges to  $AA$  for (b), (d) and  $BB$  for (c), which shows the tendency of the IGM constraint and value factorization approaches to converge to OOD joint actions. These results also suggest that the problem of joint action OOD becomes more severe when the dataset collection policy is diverse and/or the environment has multiple global optima requiring higher levels of coordination.
5.2 Bridge
Bridge is a stateful extension of the XOR Game, where two agents must take turns crossing a narrow bridge. We introduce a harder version (Figure 3) where both agents start "on the bridge" rather than on the diagonal cells of the opponent goal states as in the original game.
This subtle change makes the task much harder because now there are only two optimal actions: (Left, Left) and (Right, Right) at the initial state. Conversely, the original game can be solved optimally as long as at least one agent goes on the bridge.
Figure 3: Bridge (Hard)
The optimal dataset (500 trajectories) was constructed by a mixture of deterministic optimal policies which randomizes between Agent 1 crossing the bridge first while Agent 2 retreats, and vice-versa. The mix dataset further adds 500 trajectories by a uniform random policy.
OptiDICE can be naively extended to MARL by training a single  $\nu(s)$  network and running Weighted BC as the policy extraction procedure to learn factorized policies, which does not require learning a joint state-action value function. Still, it has some issues (Appendix C).
$^{8}$ Fu et al. [5] showed that separating policy parameters are necessary for solving challenging coordination tasks such as Bridge.
Table 2: Mean return and standard error (over 5 random seeds) on the Bridge domain.
Dataset	BC	ICQ	OMAR	MADTKD	OptiDICE	AlberDICE
Optimal	-1.26	-2.21 ± 0.90	-1.81 ± 0.12	-6.01 ± 0.00	-4.31 ± 0.27	-2.71 ± 0.69	-1.27 ± 0.03
Mix	-4.56	-5.88 ± 0.49	-6.01 ± 0.00	-6.01 ± 0.00	-6.58 ± 0.26	-1.76 ± 0.17	-1.29 ± 0.00
(a) AlberDICE  $(i = 1)$
AlberDICE  $(i = 2)$
Figure 4: Visualization of learned policies at the initial state for (from Left to Right) AlberDICE (agent 1, agent 2) and BC (agent 1, agent 2), for the Bridge (Hard) on the optimal dataset. The blue arrows indicate agent 2's policies when agent 1 is at  $\bullet$  and red arrows indicate agent 1's policy when agent 2 is at  $\bullet$ .
(c) BC  $(i = 1)$
(d) BC  $(i = 2)$
Table 3: Mean performance and standard error (over 3 random seeds) on the Warehouse domain.
Tiny (11x11)	Small (11x20)
(N=2)	(N=4)	(N=6)	(N=2)	(N=4)	(N=6)
BC	8.80 ± 0.25	11.12 ± 0.19	14.06 ± 0.32	5.54 ± 0.06	7.88 ± 0.14	8.90 ± 0.13
ICQ	9.38 ± 0.75	12.13 ± 0.44	14.59 ± 0.16	5.43 ± 0.19	7.93 ± 0.19	8.87 ± 0.22
OMAR	6.77 ± 0.64	14.39 ± 0.91	16.13 ± 1.21	4.40 ± 0.34	7.12 ± 0.38	8.41 ± 0.49
MADTKD	6.24 ± 0.60	9.90 ± 0.21	13.06 ± 0.19	3.65 ± 0.34	6.85 ± 0.36	7.85 ± 0.52
OptiDICE	8.70 ± 0.06	11.13 ± 0.44	14.02 ± 0.36	4.84 ± 0.32	7.68 ± 0.09	8.47 ± 0.26
AlberDICE	11.15 ± 0.35	13.11 ± 0.32	15.72 ± 0.36	5.97 ± 0.11	8.18 ± 0.19	9.65 ± 0.13
Table 4: Mean success rate and standard error (over 5 random seeds) on GRF
RPS (N=2)	3vs1 (N=3)	CA-Hard (N=4)	Corner (N=10)
BC	0.69 ± 0.08	0.44 ± 0.07	0.70 ± 0.07	0.24 ± 0.04
ICQ	0.53 ± 0.39	0.00 ± 0.00	0.00 ± 0.00	0.00 ± 0.00
OMAR	0.00 ± 0.01	0.01 ± 0.01	0.00 ± 0.00	0.07 ± 0.06
MADTKD	0.56 ± 0.16	0.56 ± 0.05	0.69 ± 0.05	0.32 ± 0.07
OptiDICE	0.71 ± 0.07	0.50 ± 0.05	0.67 ± 0.15	0.26 ± 0.04
AlberDICE	0.75 ± 0.12	0.59 ± 0.12	0.83 ± 0.04	0.36 ± 0.04
The performance results in Table 2 show that AlberDICE can stably perform near-optimally in both the optimal and mix datasets. Also, the learned policy visualizations for the optimal dataset in Figure 4 show that AlberDICE is the only algorithm which converges to the optimal deterministic policy in the initial state, similar to the results in the Matrix game. We also include similar policy visualization results for all states and algorithms in Appendix H.
5.3 High-Dimensional MARL Benchmarks
We further evaluate AlberDICE on standard MARL benchmarks including RWARE [26], GRF [12] and SMAC [31]. For RWARE and GRF, we train an autoregressive policy using Multi-Agent Transformers (MAT) [37] in order to collect diverse trajectories for constructing offline datasets. For SMAC, we use the public dataset provided by Meng et al. [19].
RWARE simulates a real-world warehouse in which robots move and deliver requested goods in a partially observable environment (each agent can observe the  $3 \times 3$  square centered on the agent). RWARE requires high levels of coordination, especially whenever the density of agents is high and there are narrow pathways where only a single agent can pass through (similar to Bridge).
The results in Table 3 show that AlberDICE performs on-par with OMAR in the Tiny  $(11\times 11)$  environment despite OMAR being a decentralized training algorithm. As shown in Figure 9(a) of [26], a large portion of the Tiny map contains wide passageways where agents can move around relatively freely without worrying about colliding with other agents. On the other hand, AlberDICE outperforms baselines in the Small  $(11\times 20)$  environment (shown in Figure 9(b) of [26]), where precise coordination among agents becomes more critical since there are more narrow pathways and the probability of a collision is significantly higher. We also note that the performance gap between AlberDICE and baselines is largest when there are more agents  $(N = 6)$  in the confined space. This increases the probability of a collision, and thus, requires higher levels of coordination.
Table 5: Mean success rate and standard error (over 5 random seeds) on SMAC
3s5z (Hard) (N=8)	5m_vs_6m (Hard) (N=5)	Corridor (SH) (N=6)	6hvs8z (SH) (N=6)	8m_vs_9m (Hard) (N=8)	3s5z_vs_3s6z (SH) (N=8)
BC	0.30 ± 0.05	0.23 ± 0.02	0.90 ± 0.02	0.11 ± 0.02	0.48 ± 0.05	0.45 ± 0.03
ICQ	0.18 ± 0.08	0.18 ± 0.10	0.78 ± 0.03	0.00 ± 0.00	0.12 ± 0.21	0.31 ± 0.04
OMAR	0.43 ± 0.04	0.18 ± 0.02	0.92 ± 0.02	0.15 ± 0.03	0.45 ± 0.05	0.60 ± 0.05
MADTKD	0.12 ± 0.02	0.19 ± 0.02	0.67 ± 0.01	0.09 ± 0.02	0.14 ± 0.04	0.18 ± 0.02
OptiDICE	0.28 ± 0.05	0.21 ± 0.02	0.91 ± 0.02	0.13 ± 0.00	0.47 ± 0.05	0.42 ± 0.04
AlberDICE	0.47 ± 0.03	0.24 ± 0.03	0.98 ± 0.00	0.21 ± 0.03	0.67 ± 0.06	0.63 ± 0.03
(a) Histogram of uncertainty estimates  $\{U(s,\mathbf{a}):s\in D,a\sim \pi (s)\}$
Figure 5: Experimental results to see how effectively AlberDICE avoids OOD joint actions.
8m_vs_9m	3s5z_vs_3s6z
(N=8)	(N=8)
BC	0.1%	0.3%
ICQ	54.7%	26.2%
OMAR	16.5%	5.7%
MADTKD	1.7%	0.8%
OptiDICE	0.1%	0.3%
AlberDICE	4.8%	1.6%
(b) Percentage of selecting OOD joint actions.
Our results for GRF and SMAC in Tables 4 and 5 show that AlberDICE performs consistently well across all scenarios, and outperforms all baselines especially in the Super Hard maps, Corridor and  $6\mathrm{h8z}$ . The strong performance by AlberDICE corroborates the importance of avoiding OOD joint actions in order to avoid performance degradation.
5.4 Does AlberDICE reduce OOD joint actions?
In order to evaluate how effectively AlberDICE prevents selecting OOD joint actions, we conducted additional experiments on two SMAC domains (8m_vs_9m and 3s5z_vs_3s6z) as follows. First, we trained an uncertainty estimator  $U(s,\mathbf{a})$  via fitting random prior [2]  $f:S\times \mathcal{A}_1\times \dots \times \mathcal{A}_N\to \mathbb{R}^m$  using the dataset  $D = \{(s,\mathbf{a})_k\}_{k = 1}^{|D|}$ . Then,  $U(s,\mathbf{a}) = \| f(s,\mathbf{a}) - h(s,\mathbf{a})\| ^2$  outputs low values for in-distribution  $(s,\mathbf{a})$  samples and outputs high values for out-of-distribution  $(s,\mathbf{a})$  samples. Figure 5(a) shows a histogram of uncertainty estimates  $U(s,\pi_1(s),\ldots ,\pi_N(s))$  for each  $s\in D$  and the joint action selected by each method. We set the threshold  $\tau$  for determining OOD samples to  $99.9\%$ -quantile of  $\{U(s,a):(s,a)\in D\}$ . Figure 5(b) presents the percentage of selecting OOD joint actions by each method. AlberDICE selects OOD joint actions significantly less often than ICQ (IGM-based method) and OMR (decentralized training method) while outperforming them in terms of success rate (see Table 5).
DICE for Offline RL Numerous recent works utilize the LP formulation of RL to derive DICE algorithms for policy evaluation [20-22]. OptiDICE [14] was introduced as the first policy optimization algorithm for DICE and as a stable offline RL algorithm which does not require value estimation of OOD actions. While OptiDICE can be naively extended to offline MARL in principle, it can still fail to avoid OOD joint actions since its primary focus is to optimize over the joint action space of the MMDP and does not consider the factorizability of policies. We detail the shortcomings of a naive extension of OptiDICE to multi-agent settings in Appendix C.
Value-Based MARL A popular method in cooperative MARL is (state-action) value decomposition. This approach can be viewed as a way to model  $Q(s, a)$  implicitly by aggregating  $Q_{i}$  in a specific manner, e.g., sum [34], or weighted sum [30]. Thus, it avoids modelling  $Q(s, a)$  explicitly over the joint action space. QTRAN [33] and QPLEX [36] further achieve full representativeness of IGM
9. These approaches have been shown to perform well in high-dimensional complex environments including SMAC [31]. However, the IGM assumption and the value decomposition structure have been shown to perform poorly even in simple coordination tasks such as the XOR game [5].
Policy-Based MARL Recently, policy gradient methods such as MAPPO [41] have shown strong performance on many complex benchmarks including SMAC and GRF. Fu et al. [5] showed that independent policy gradient with separate parameters can solve the XOR game and the Bridge environment by converging to a deterministic policy for one of the optimal joint actions. However, it requires an autoregressive policy structure (centralized execution) to learn a stochastic optimal policy which covers multiple optimal joint actions. These empirical findings are consistent with theoretical results [15, 44] showing that running independent policy gradient can converge to a Nash policy in cooperative MARL. On the downside, policy gradient methods are trained with on-policy samples and thus, cannot be extended to the offline RL settings due to the distribution shift problem [16].
Offline MARL ICQ [40] was the first MARL algorithm applied to the offline setting. It proposed an actor-critic approach to overcome the extrapolation error caused by the evaluation of unseen state-action pairs, where the error is shown to grow exponentially with the number of agents. The centralized critic here uses QMIX [30] and thus, it inherits some of the weaknesses associated with value decomposition and IGM. OMAR [25] is a decentralized training algorithm where each agent runs single-agent offline RL over the individual Q-functions and treats other agents as part of the environment. As a consequence, it lacks theoretical motivation and convergence guarantees in the underlying MMDP or Dec-POMDP. MADTKD [35] extends Multi-Agent Decision Transformers [19] to incorporate credit assignment across agents by distilling the teacher policy learned over the joint action space to each agent (student). This approach can still lead to OOD joint actions since the teacher policy learns a joint policy over the joint action space and the actions are distilled individually to students.
AlberDICE relies on Nash policy convergence which is a well-established solution concept in Game Theory, especially in the general non-cooperative case where each agent may have conflicting reward functions. One limitation of AlberDICE is that the Nash policy may not necessarily correspond to the global optima in cooperative settings. The outcome of the iterative best response depends on the starting point (region of attraction of each Nash policy) and is, thus, generally not guaranteed to find the optimal Nash policy [1]. This is the notorious equilibrium selection problem which is an open problem in games with multiple equilibria, even if they have common reward structure (See Open Questions in [15]). Nonetheless, Nash policies have been used as a solution concept for iterative update of each agents as a way to ensure convergence to factorized policies in Cooperative MARL [10]. Furthermore, good equilibria tend to have larger regions of attraction and practical performance is typically very good as demonstrated by our extensive experiments.
In this paper, we presented AlberDICE, a multi-agent RL algorithm which addresses the problem of distribution shift in offline MARL by avoiding both OOD joint actions and the exponential nature of the joint action space. AlberDICE leverages an alternating optimization procedure where each agent computes the best response DICE while fixing the policies of other agents. Furthermore, it introduces a regularization term over the stationary distribution of states and joint actions in the dataset. This regularization term preserves the common reward structure of the environment and together with the alternating optimization procedure, allows convergence to Nash policies. As a result, AlberDICE is able to perform robustly across many offline MARL settings, even in complex environments where agents can easily converge to sub-optimal policies and/or select OOD joint actions. As the first DICE algorithm applied to offline MARL with a principled approach to curbing distribution shift, this work provides a starting point for further applications of DICE in MARL and a promising perspective in addressing the main problems of offline MARL.
<sup>9</sup>Details about IGM are provided in A.1
This work was partly supported by the IITP grant funded by MSIT (No.2020-0-00940, Foundations of Safe Reinforcement Learning and Its Applications to Natural Language Processing; No.2022-0-00311, Development of Goal-Oriented Reinforcement Learning Techniques for Contact-Rich Robotic Manipulation of Everyday Objects; No.2019-0-00075, AI Graduate School Program (KAIST); No.2021-0-02068, AI Innovation Hub), NRF of Korea (NRF2019R1A2C1087634), Field-oriented Technology Development Project for Customs Administration through NRF of Korea funded by the MSIT and Korea Customs Service (NRF2021M3I1A1097938), KAIST-NAVER Hypercreative AI Center, the BAIR Industrial Consortium, and NSF AI4OPT AI Centre.