Title: Meta-learning with Auto-generated Tasks for Predicting Human Behaviour in Normal Form Games
Authors: Shaofei Chen
Jiaxing Chen
Peng Li
William Yuan
Zhen Zhen Hu
Venue: ICLR 2023

Abstract: In recent years, machine learning methods have been successfully applied to predict human behaviour in strategic settings. However, as available data of human behaviour is not always large enough and people's reasoning processes in different types of games are various, it is challenging to acquire a satisfied prediction model. In this paper, we employ a meta-learning method to improve learning performance in predicting human behaviour in normal form games. In particular, we first design a deep neural network that captures mixed human behaviour features to model and be learned to get a underlying behavioural predictor. Then, using a dataset of experimental human behaviour, we apply unsupervised learning to generate tasks and use meta-learning to improve the learning proficiency. Experimental results show that our proposed meta-learning method with the designed neural network and auto-generated tasks considerably increases the prediction accuracy and significantly exceeds the previous state-of-the-art.


Review discussion:
Strengths:
* The paper is well organized and cites the related work.
Weakness:
* The  paper can be hard to follow at certain sections, concepts could be explained clearly
* The significance of the work is a bit unclear. There doesn't appear to be significant algorithmic novelty using the existing MAML approach.
* While the experimental evaluation results seem promising (Section 5.2) the real world significance could be motivated further, ie performance on more complex games mimicking real world scenarios.
Nit Section 4.3: 'methoud'-> 'method'
Overall, the work seems reasonably novel. The authors have provided sufficient evidence in Section 4 for this to be reproducible. The dataset has been designed for this study, more details would be helpful.
NO.
N/A
We thank the reviewer’s constructive feedback and answer specific questions below:
> The paper can be hard to follow at certain sections, concepts could be explained clearly.
We have rewritten the paper (especially Figure 1, Section 4, 5 and Appendix) and polished the sentences.
> The significance of the work is a bit unclear. There doesn't appear to be significant algorithmic novelty using the existing MAML approach.
It is true that the novelty of this paper is not using the existing MAML approach.
We have rewritten the key sentences in this paper to highlight the motivation and the novelty, which is as follows:
* We
designed and verified a deep neural network that embodies mixed human behaviour features
that follows cognitive psychology theory, which outperforms FAR, a neural network with multiple levels of hidden matrix units and pooling units.
* The novelty of our meta-learning method is on tasks that generated by
clustering the extracted mixture of features of human behaviour and the experiments results have proven its performance.
> The dataset has been designed for this study, more details would be helpful.
We have added more details of the network structure and highlighted the learnable model parameters in Section 4 and Appendix.
We have added the details of the dataset in Appendix Section A.3.
## Meta-learning
I do not understand the motivation for meta learning in this particular case where the whole dataset is known a priori.
The goal of MAML is to discover a good initialization point for the network such that it can adapt quickly to any new task that may be present at test time. In this case, all the tasks are known at train time, could you please clarify why Meta Learning is useful?
The benefit of the Meta-learning formulation doesn't seem to be very well established empirically either. If we eye-ball figure 5.3, left, the test loss of the MFAR model is roughly the same as the best MetaLearning one (MF4), even though the comparison is not fair: as far as I understand MFAR is trained for 2.5k steps while MF5 is effectively trained for 41k steps (800 meta epochs * 50 updates + 1000 "adaptations steps")
I do not understand what "evaluated by using one gradient step at test time with the best prediction performance" means, does that imply that the method is trained on the test set?
## FAR vs MFAR
I am surprised to see a big difference between the test NLL reported in this paper for the FAR (50,50) model (around 1600 according to Fig 5.1.a) compared to the original paper [1], which reports numbers closer to 950. Explanations in the differences would be welcomed.
As for MFAR, the description of the model is not clear to me. I gather that the "experts" are designed to compute handcrafted features, but do they retain any learnable parameters? In general, reporting the number of trainable parameters in both models would be valuable.
[1] "Deep Learning for Predicting Human Strategic Behavior", Hartford et al, Neurips 2016
The paper is generally hard to read and poorly motivated. It contains many typos (see at the end of this section for a sample)
Typos:
- "Five non-strategic huamn"
- "acquire a satisfied prediction model"
- "that proper to be used"
- " depends with the input"
- "adaptive aoment astimation"
- "our methoud"
- "meta-itrations"
NO.
We thank the reviewer’s constructive feedback and answer specific questions below:
> the motivation for meta learning.
As people's reasoning processes in different types of games are various, the dataset can be seen
as containing small samples from different domains.
The main motivation for meta-learning is to improve the learning performance on this kind of dataset.
The previous manuscript is not clear on this and we have highlighted this and added the experiments' results on
generalisation Performance in Section 5.2.
> The benefit of the Meta-learning formulation doesn't seem to be very well established empirically either..
We have updated the experimental results of MFAR-ML and shown them in Table 1. MFAR-ML's 745 is 6.3% better than MFAR's 795, and  FAR-ML's 939 is 37% better than FAR's 1497.
Moreover, we have added experiments to validate these methods' generalizability to new
games using 80% trained and 20% validated proportional datasets. The results are shown in Table 2 and the final paragraph of Section 5.2.
> a big difference between the test NLL reported in this paper for the FAR (50,50) model (around 1600 according to Fig 5.1.a) compared to the original paper.
We used the open source code provided in the (Hartford et al., 2016), but collated a slightly different dataset. The payoff matrices of 2×4 in (Costa-Gomes et al., 1998) and (Goeree & Holt, 2001) are not included, hence the FAR model on our dataset performs differently from how it did in the origin work. We divided the payoff matrix of both sides of a game into two for our dataset in (Cooper and Huyck, 2003), which increased the number of games. The dataset comparison between the FAR and MFAR models is added in Table 3.
> As for MFAR, the description of the model is not clear to me. I gather that the "experts" are designed to compute handcrafted features, but do they retain any learnable parameters? In general, reporting the number of trainable parameters in both models would be valuable..
The experts did not retain any learnable parameters, only the gating network and the meta-learning component had learnable parameters.
We have added this at the end of the paragraph introducing mixture of features.
A comparison of the network information architectures of the two models, FAR and MFAR, is added in Table 4.
> The paper is generally hard to read and poorly motivated. It contains many typos (see at the end of this section for a sample)
We have rewritten the paper (especially Figure 1, Section 4, 5 and Appendix) and polished the sentences. We will invite native speakers to polish the paper later.
> The contribution of the mixture of expert seems to rely on handcrafting features, which reduces its generality.
The features in FAR is modelled by multiple levels of hidden matrix units and pooling units in neural network, which performs worse than MFAR's handcrafting features as shown in our results. These features are based on cognitive psychology theory, and we take these results  to mean that the general neural network structure is inferior to the neural network based on cognitive psychological theory in this problem.
Similar conclusions have also been drawn from studies on risk choice problem [1].
[1] Joshua C. Peterson and David D. Bourgin and Mayank Agrawal and Daniel Reichman and Thomas L. Griffiths, Using large-scale experiments and machine learning to discover theories of human decision-making, Science, {372}, {6547}, {1209-1214}, 2021, doi = 10.1126/science.abe2629
Strenths:
- the authors clearly communicate their main contributions and provide the background knowledge to put them into perspective
- The problem is well motivated and the introduction provides the reader with enough information to understand the problem and solution approach
Weaknesses:
- The paper overall is very vague
- Figure 1 is providing me with a full understanding of the approach.
While it is clear to see that the constructed tasks in (B) serve as inputs in (c), I do not know where the underlying predictor from (a) comes into play.
- In section 4.1 the authors declare that: "Moreover, given its neural network architecture, MFAR is
more general than the model proposed in (Wright & Leyton-Brown, 2019) by taking weighted linear
or logit specification to ensemble the non-strategic behaviours." The authors essentially introduce additional parameters (that are learned? I am not sure at is it never mentioned). If these parameters are learned, then this would increase the amount of data needed to train the model - this is a problem that should should be discussed as the authors explicitly mention a limited data set
- In section 4.2 the autors mention the use of an autoencoder.
However, there is no information about what kind of autoencoder or architecture details. This significantly reduces understanding and reproducability for the experiments.
- There are several vague and borderline wrong statements in section 4.3. One example is " The loss LTt(Pθ ) is defined by a certain function to evaluate predicting performance on task Tt with model Pθ .
While this is certainly a true statement, it does not allow for any insights into the chosen loss function (which is not reported anywhere else).
Another example is "Through a meta-learning process that gains knowledge from a large number of behaviour prediction tasks with little data, ..." gaining knowledge from a large number of tasks with little data seems like a very counter-intuive statement.
"Hence, we make use of MAML, which initialises the weights of a network such that adaptive aoment astimation (Adam) can make rapid progress on a new task" While MAML does provide an advantagous initialization, it does NOT ensure or enable Adam to make rapid progress. The authors seem to confuse rapid SGD (which is the optimizer) with the initialization close(r) to the minimum.
- While the autors talk and show training and test error, they do not clearly state how the data is split up. They furthermore do not address the potential issue of their multi-stage process where training data could bleed into the test data via the meta-learning approach.
- Figure 5.1: These results seem very off to me.
While the learning curves look reasonable to me, the test curves are very unexpected. Especially the learning curves for MFAR (50,50) and MFAR (50,50,5) look suspiciously like there are problems with the training and test dataset.
- Figure 5.1:  Except for (a), none of the test loss results are conclusive. The curves are mostly overlapping. I am also missing a statistical analysis of the results.
- Task construction in Section 5.2.: I am missing a quantitative analysis of the results. In figure 5.2. (d) there seems to be a substantial overlap between the classes 1 and 2 in the upper right corner, so without any numerical proof, it is hard to follow the conclusion that this is the best clustering approach.
- Figure 5.3. The axis of both graphs are off by an order of magnitude. This is highly confusing and misleading.
Clarity:
The paper is clearly missing clarity. It is unclear to me how the different contributions are interacting with each other (see weakness for Figure 1). There are numerous vague and hand-wavy statements in the methods section.
Quality:
There are several typos and sentences with poor grammar (e.g.
However, statistics in (Ho et al., 2004)  how that cognitive levels many people are more than one., adaptive aoment astimation). The paper would significantly benefit from careful proofreading.
Novelty:
The main novelty of the papers seems to be the adjustable gating units in the prediction model, paired with the novel use of clustering and meta-learning.
While the (to my knowledge) combination of the three appraoches is novel in this field, the adjustable gating units are (in my opinion) an incremental step. Especially as the authors do not address the potential downsides when it comes to limited training data.
NO.
We thank the reviewer’s constructive feedback and answer specific questions below:
> It’s not clear in Figure 1 that where the underlying predictor from (a) comes into play.  Figure 1 only connects the clustering and the meta-learner. It is unclear how they interact with the prediction model.
We have redrawn Figure 1 to clearly depict the connection between the underlying predictor and the clustering and modified the corresponding sentences in Section 1 and 4.
> Additional parameters are introduced in MFAR. . If these parameters are learned, then this would increase the amount of data needed to train the model
In our proposed model, no other parameters are introduced in the expert part of the hybrid expert layer, and the parameters are only present in the gated network part. Specific information on the model parameters is given in Appendix Table 4.
> The details of the used autoencoder is not clear enough
The automatic encoder is used to extract the main characteristics of the return matrix of both sides of the game. The network architecture and parameter information are shown in Table 5.
> There are several vague and borderline wrong statements in section 4.3
We have carefully checked Section 4.3, revised or removed some inaccurate and/or unnecessary descriptions.
> The split up of the training data and test data. And the data are used in the multi-stage process
As the reviewer mentioned, the training data did bleed into the test data. In the validation phase, we randomly disrupted the order of the whole dataset and used 10-fold cross-validation, keeping in line with the original paper (Hartford et al., 2016).
Based on the clustering results, the 127 data were divided into n classes, and in each subclass, the dataset was randomly divided into ten copies, with nine copies of the training set and one copy of the test set taken for each meta-training. In the validation phase, the order of the whole dataset was randomly disrupted and 10-fold cross-validation was used to test the effect of meta-training.
> Figure 5.1: These results seem very off to me.
While the learning curves look reasonable to me, the test curves are very unexpected. Especially the learning curves for MFAR (50,50) and MFAR (50,50,5) look suspiciously like there are problems with the training and test dataset. Figure 5.1: Except for (a), none of the test loss results are conclusive. The curves are mostly overlapping. I am also missing a statistical analysis of the results.
The learning curves in Figure 5.1 do not use meta-learning, but instead use the Mixture of Features architecture model compared to the training process of the original paper to demonstrate the rapidity and effectiveness of the Mixture of Features architecture learning. As our dataset is only 127 data, which is a small sample problem, it is prone to overfitting problems of neural networks, and to address this problem, we used meta-learning to improve the model based on the Mixture of Features architecture.
> Task construction in Section 5.2.: I am missing a quantitative analysis of the results. In figure 5.2(d) there seems to be a substantial overlap between the classes 1 and 2 in the upper right corner, so without any numerical proof, it is hard to follow the conclusion that this is the best clustering approach.
We have added more experiments' results in Section 5.2 and Appendix, especially a quantitative analysis of clustering results in Appendix 6.
> Figure 5.3. The axis of both graphs are off by an order of magnitude. This is highly confusing and misleading..
We have changed Figure 5.3 to Table 1. This makes the results clearer.
> There are several typos and sentences with poor grammar.
We have rewritten the paper (especially Section 4, 5 and Appendix) and polished the sentences. We will invite native speakers to polish the paper later.
Strengths
• The paper addresses an interesting problem of predicting human behavior in a well-defined setting
• The figures are clear and informative
Weaknesses
• The paper makes a number of assumptions that don't hold generally.
For example, "a beneficial common initialisation would serve as a good point to start training for a new set of games being considered" applies only if the new games being considered are able to be solved by a model that's close enough in parameter space to the MAML prior so that the model performs well on the new games with few gradient updates (i.e., few-shot learning)
• The first research question is "Can the MoE architecture in MFAR improve the learning accuracy?" MFAR is a superset of FAR, so there's no doubt that it "can" and almost certainly will improve performance. This isn't a research question as the answer is already known--yes.
• The second research question again is not research question as the answer is known--yes.
• The results in Figure 5.1 don't make sense. Why does the FAR(50, 50, 5), the most highly parametrized FAR model perform worse than the other two FAR models on the training set? It should obtain lower training loss than FAR(50, 50) and FAR(5, 5) since all of these are trained for the same number of iterations.
• t-SNE visualizations are brittle to parameters in the t-SNE function, so they are not convincing in assessing qualitative clustering quality. PCA projections are an example of visualizations
• There's only one baseline class being compared against, FAR.
• FAR is a weak baseline to compared against because it is a subset of MFAR.
Comments
The paper mentions "auto-encoder network" on page 6 and cites Michelucci (2022). Please reference work(s) that first introduced autoencoders, such as Rumelhart et al. (1985).
The Figure 5.1 has very thin red and green lines and faint shading, which are not red-green colorblind-friendly. Please consider using colorblind-friendly colors instead, especially for thin lines or faint shading.
Michelucci, Umberto. "An Introduction to Autoencoders." arXiv preprint arXiv:2201.03898 (2022).
Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. Learning internal representations by error propagation. California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
The paper is clear in its exposition, but becomes unclear afterwards, makes assumptions that are too strong, and has results that don't make sense. The quality of the work is low for reasons mentioned above, some of which include wholly poor experimental design--only one baseline class and the baseline is a poor quality one, and the novelty is almost non-existent as the model is simply MoE (existing and common) + FAR (existing) + MAML. The work seems reproducible as the algorithm is simple and the paper does provide hyperparameters and other relevant details.
NO.
We thank the reviewer’s constructive feedback and answer specific questions below:
> The assumption applies to a model that's close enough in parameter space to the MAML prior.
It's true that some descriptions of meta-learning's role in the previous manuscript are not accurate. We have fully checked and revised the these such as the content in Section 4.3.
> The first two research questions are not research questions.
* Compared to FAR, MFAR not only adds a gating structure, but also specifies the structure of each expert based on cognitive psychology theory.
Although there's no doubt that it "can" and almost certainly will improve performance, we think it is still interesting to verify how much this approach improves the prediction performance.
* Although the effect of meta-learning has been demonstrated in many problems, our problem is different because the tasks are constructed by our designed methods.The experimental results has also proved this. As we can see in Table 1, despite the use of meta-learning, the performances of MFAR-ML(D), MFAR-ML(NE)  and MFAR-ML(DC) are not improved compared with that of MFAR. The superiority of MFAR-ML(MF) over MFAR verifies the validity of the task construction method with MF.
> The FAR(50, 50, 5) performs worse than FAR(50, 50).
With a small sample size of only 127 in the dataset, the higher the model complexity, the more parameters there are. It is difficult to train more complex models with small samples, which is a limitation of deep learning. In the experiments in (Hartford et al., 2016), the less parameterised FAR (50,50) was also chosen as the final model instead of FAR (50, 50, 5).
> t-SNE visualizations are brittle to parameters. PCA projections are an example of visualizations.
As the reviewer said, t-SNE needs to set parameters.
However, we conducted ablation experiments on this parameter, and used different parameter values with t-SNE to visualise the data, and the results were consistent with those in the paper, as shown in Figure A.3.
Furthermore, t-SNE is a nonlinear dimension reduction method, which is very suitable for high-dimensional data visualisation, such as literature [1,2].
However, PCA is a linear dimension reduction method, which can not explain the complex polynomial relationship between features, as shown in Table 6.
[1] Joshua C. Peterson and David D. Bourgin and Mayank Agrawal and Daniel Reichman and Thomas L. Griffiths, Using large-scale experiments and machine learning to discover theories of human decision-making, Science, {372}, {6547}, {1209-1214}, 2021, doi = 10.1126/science.abe2629
[2] J. Chang, G. Meng, L. Wang, S. Xiang and C. Pan, "Deep Self-Evolution Clustering," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 4, pp. 809-823, 1 April 2020, doi: 10.1109/TPAMI.2018.2889949.
> There's only one baseline, FAR. FAR is a weak baseline to compared against because it is a subset of MFAR.
In (Hartford et al., 2016), FAR has been shown to outperform other baselines from cognitive psychology theory, and our MFAR performs far better than FAR.
There is no more baseline that currently need to compare.
> The citation of "auto-encoder network" should be changed.
We have changed this citation according to the reviewer.
> The lines in Figure 5.1 are not red-green colorblind-friendly.
The color matching has been changed to colour-blind friendly color.
We thank all the reviewers for their constructive feedback and suggestions.
On the whole, the reviewers pointed out that paper is not clear, and the setting and analysis of experiments were not sufficient. In this version, We have rewritten the paper (especially Figure 1, Section 4, 5 and Appendix) and polished the sentences. Due to the limited time for rebuttal, we will invite native speakers to further polish the paper later.
The main additions are summarised below:
* We have redrawn Figure 1 to clearly depict the connection between the underlying predictor, clustering and meta-learning, and modified the corresponding sentences in Section 1 and 4.
* We have added experiments to validate these methods' generalizability to new games using 80% trained and 20% validated proportional datasets. The results are shown in Table 2 and the final paragraph of Section 5.2.
* We have added some statistical analysis of experimental results in Section 5 and Appendix.
* We have added the details of the hyper-parameters of the deigned neural networks and used auto-encoder in Table 4 and 5.
* we have modified and highlighted the motivation of using meta-learning. We have also highlighted our contribution and novelty in corresponding position such as in introduction and related works.

Meta-review:
role: PC
data: metareview:_summary,_strengths_and_weaknesses: (a) Summary:  This paper combines a meta-learning approach with a previous DNN model for predicting human behavior in normal form games.

(b) Strengths: All reviewers agreed that the problem was well motivated and well described.

(c) Weaknesses: All reviewers noted serious clarity issues, many of which were addressed in a revision.  Multiple reviewers pointed to figure 5.1 as a particular problem (the most highly-parameterized baseline had worse performance on the training data than nested models, indicating inadequate optimization).  One reviewer had strong concerns about leakage between the training and validation datasets. Multiple reviewers noted that the contributions of the paper are not expressed sufficiently clearly (e.g., how the various parts of the algorithm relate to one another, and where the performance improvement comes from).

justification_for_why_not_higher_score: The reviewers were unanimous that this paper is not ready in its current form, and I agree.
justification_for_why_not_lower_score: n/a
summary_of_AC-reviewer_meeting: n/a

Decision: Reject