SlowGuess/NeurIPS_2023_OCR huggingface dataset:
def get triplet msgs_v1(z, edge_fts, graph_fts, nb triplet_fts, out_size):  
    node_reps = [hk.Linear(nb triplet_fts) for _ in range(3)]  
    edge_reps = [hk.Linear(nb triplet_fts) for _ in range(3)]  
    graph_rep = hk.Linear(nb triplet_fts)  
    triplet_node_reps = [node_rep(z) for node_rep in node_reps]  
    triplet_edge_reps = [edge_rep(edge_fts) for edge_rep in edge_reps]  
    triplet_graph_rep = graph_rep/graph_fts)  
    node_pair.permutations = [(2, 3), (1, 3), (1, 2)]  
    triplets = functools.reduce()  
    lambda x, y: x + y,  
    [  
        jnp expand_dims(tri_node_rep, axis=perm)  
        for tri_node_rep, perm in zip(  
            triplet_node_reps, node_pair.permutations  
        )  
    ],  
    ]  
    triplets += functools.reduce()  
    lambda x, y: x + y,  
    [  
        jnp expand_dims(tri_edge_rep, axis=i)  
        for tri_edge_rep, i in zip(triplet_edge_reps, range(3, 0, -1))  
        ],  
    ]  
    triplets += jnp expand_dims(triplet_graph_rep, axis=(1, 2, 3))  
    output_layer = hk.Linear(out_size)  
    return output_layer(jnp.max(triplets, axis=1))
Listing 11: The triplet representation belonging to the first seed model - the standard triplet representation from Ibarz et al. (2022).
def get triplet msg s v2 (z, edge_fts, graph_fts, nb triplet_fts, out_size):   
node_reps  $=$  [hk.Linear(nb_triplet_fts) for in range(3)]   
edge_reps  $=$  [hk.Linear(nb_triplet_fts) for in range(3)]   
graph_rep  $=$  hk.Linear(nb_triplet_fts)   
triplet_node_reps  $=$  [node_rep(z) for node_rep in node_reps]   
triplet_edge_reps  $=$  [edge_rep(edge_fts) for edge_rep in edge_reps]   
triplet_graph_rep  $=$  graph_rep/graph_fts)   
node_pair.permutations  $= [(2,3),(1,3),(1,2)]$    
triplets  $=$  functools.reduce( lambda x,y:  $\mathbf{x} + \mathbf{y}$  1 [ jnp.expand_dims(tri_node_rep, axis  $\equiv$  perm) for tri_node_rep,perm in zip( triplet_node_reps, node_pair.permutations ] ],   
1   
1   
1   
1   
1   
1   
1   
1   
1   
1   
1   
1   
1   
1   
1   
1   
1   
1   
1   
1   
1
23 ]，   
24 ）   
25 triplet  $\text{十} =$  jnp.exp_dims(triplet_graph_rep，axis=(1，2，3))   
26 output_layer  $=$  hk.nets.MLP([out_size，out_size])   
27 return output_layer(jnp.max(triplets，axis=1))
Listing 12: The triplet representation belonging to the second seed model, with the output layer replaced by a fully-connected multi-layer perceptron.
def get triplet msgs_v3(z, edge_fts, graph_fts, nb triplet_fts, out_size):  
    node_reps = [hk.Linear(nb triplet_fts) for _ in range(3)]  
edge_reps = [hk.Linear(nb triplet_fts) for _ in range(3)]  
graph_rep = hk.Linear(nb triplet_fts)  
triplet_node_reps = [node_rep(z) for node_rep in node_reps]  
triplet_edge_reps = [edge_rep(edge_fts) for edge_rep in edge_reps]  
triplet_graph_rep = graph_rep/graph_fts)  
node_pair.permutations = [(2, 3), (1, 3), (1, 2)]  
triplets = functools.reduce()  
lambda x, y: x + y,  
[  
    jnp.exp Expand_dims(tri_node_rep, axis=perm)  
    for tri_node_rep, perm in zip(  
        triplet_node_reps, node_pair.permutations  
    )  
],  
)  
triplets += functools.reduce()  
lambda x, y: x + y,  
[  
    jnp.exp Expand_dims(tri_edge_rep, axis=i)  
    for tri_edge_rep, i in zip(triplet_edge_reps, range(3, 0, -1))  
    ],  
)  
triplets += jnp.exp Expand_dims(triplet_graph_rep, axis=(1, 2, 3))  
output_layer = hk.Linear(out_size)  
return output_layer(jnp.sum(triplets, axis=1))
Listing 13: The triplet representation belonging to the third seed model, which uses sum instead of max aggregation.
def get triplet msgs_v4(z, edge_fts, graph_fts, nb triplet_fts, out_size):  
    node_reps = [hk*nets.MLP([nb triplet_fts, nb triplet_fts]) for _ in range(3)]  
edge_reps = [hk.Linear(nb triplet_fts) for _ in range(3)]  
graph_rep = hk.Linear(nb triplet_fts)  
triplet_node_reps = [node_rep(z) for node_rep in node_reps]  
triplet_edge_reps = [edge_rep(edge_fts) for edge_rep in edge_reps]  
triplet_graph_rep = graph_rep/graph_fts)  
node_pair_permutations = [(2, 3), (1, 3), (1, 2)]  
triplets = functools.reduce()  
lambda x, y: x + y,  
[  
    jnp.exp Expand_DIMs(tri_node_rep, axis=perm)  
    for tri_node_rep, perm in zip(  
        triplet_node_reps, node_pair_permutations  
    )
16 ],   
17 ）   
18 triplets  $+=$  functools.reduce(   
19 lambda x, y: x + y,   
20 [   
21 jnp.expands(tri_edge_rep, axis=i)   
22 for tri_edge_rep, i in zip(triplet_edge_reps, range(3,   
23 0,-1))   
24 ]，   
25 triplets  $+=$  jnp.expands(triplet_graph_rep, axis=(1,2,3))   
26 output_layer  $\equiv$  hk.Linear(out_size)   
27 return output_layer(jnp.max(triplets, axis=1))
Listing 14: The triplet representation of the 4th seed model, which uses fully-connected multi-layer perceptron node representations.
def get triplet msgs_v5(z, edge_fts, graph_fts, nb_triplet_fts, out_size):  
    node_reps = [hk.Linear(nb_triplet_fts) for _ in range(3)]  
    edge_reps = [hk=nets.MLP([nb_triplet_fts, nb Triplet_fts]) for _ in range(3)]  
    graph_rep = hk.Linear(nb_triplet_fts)  
    triplet_node_reps = [node_rep(z) for node_rep in node_reps]  
    triplet_edge_reps = [edge_rep(edge_fts) for edge_rep in edge_reps]  
    triplet_graph_rep = graph_rep/graph_fts)  
    node_pair.permutations = [(2, 3), (1, 3), (1, 2)]  
    triplets = functools.reduce()  
    lambda x, y: x + y,  
        [  
            jnp.expand_dims(tri_node_rep, axis=perm)  
            for tri_node_rep, perm in zip(  
                triplet_node_reps, node_pair.permutations  
                )  
            ],  
        ]  
    triplets += functools.reduce()  
    lambda x, y: x + y,  
        [  
            jnp.expand_dims(tri_edge_rep, axis=i)  
            for tri_edge_rep, i in zip(triplet_edge_reps, range(3, 0, -1))  
            ],  
        ]  
    triplets += jnp.expand_dims(triplet_graph_rep, axis=(1, 2, 3))  
    output_layer = hk.Linear(out_size)  
    return output_layer(jnp.max(triplets, axis=1))
Listing 15: The triplet representation of the 5th seed model, which uses fully-connected multi-layer perceptron edge representations.
def get triplet msgs_v6(z, edge fps, graph fps, nb triplet fps, out_size):  
    node_reps = [hk.Linear(nb triplet fps) for _ in range(3)]  
    edge_reps = [hk.Linear(nb triplet fps) for _ in range(3)]  
    graph_rep = hk.nets.MLP([nb triplet fps, nb triplet fps])  
    triplet_node_reps = [node_rep(z) for node_rep in node_reps]  
    triplet_edge_reps = [edge_rep(edge fps) for edge_rep in edge_reps]  
    triplet_graph_rep = graph_rep/graph fps
node_pair.permutations  $=$  [(2,3)，(1，3)，(1，2)]   
triplets  $=$  functools.reduce( lambda x,y:  $\mathbf{x} + \mathbf{y}$  [ jnp.expand_dims(tri_node_rep，axis  $\equiv$  perm) for tri_node_rep，perm in zip( triplet_node_reps，node_pair.permutations 1]，   
）   
triplets  $= =$  functools.reduce( lambda x,y:  $\mathbf{x} + \mathbf{y}$  [ jnp.expand_dims(tri_edge_rep，axis=i) for tri_edge_rep，i in zip(triplet_edge_reps，range(3, 0,-1)) ]，   
）   
triplets  $= =$  jnp.expand_dims(triplet_graph_rep，axis=(1，2，3)) output_layer  $=$  hk.Linear(out_size) return output_layer(jnp.max(triplets，axis=1))
Listing 16: The triplet representation of the 6th seed model, which uses fully-connected multi-layer perceptron graph representations.
def get triplet msgs_v7(z, edge_fts, graph_fts, nb triplet_fts, out_size):  
    node_reps = [hk*nets.MLP([nb_triplet_fts]) for _ in range(3)]  
    edge_reps = [hk.Linear(nb_triplet_fts) for _ in range(3)]  
    triplet_node_reps = [node_rep(z) for node_rep in node_reps]  
    triplet_edge_reps = [edge_rep(edge_fts) for edge_rep in edge_reps]  
    node_pair.permutations = [(2, 3), (1, 3), (1, 2)]  
    triplets = functools.reduce( lambda x, y: x + y, [ jnp expand_dims(tri_node_rep, axis=perm) for tri_node_rep, perm in zip( triplet_node_reps, node_pair.permutations ] ],  
    triplets += functools.reduce( lambda x, y: x + y, [ jnp expand_dims(tri_edge_rep, axis=i) for tri_edge_rep, i in zip(triplet_edge_reps, range(3, 0, -1)) ],  
    output_layer = hk.Linear(out_size)  
    return output_layer(jnp.max(tripllets, axis=1))
Listing 17: The triplet representation of the 7th seed model, which uses fully-connected multi-layer perceptron node representations and does not have a graph representation.
def get triplet msgs_v8(z, edge_fts, graph_fts, nb triplet_fts, out_size):  
    output_layer = hk.nets.MLP([out_size])  
    return jnp.tile()
8 9   
10   
11   
12   
13   
14   
15   
16   
17   
18   
19   
20   
21   
22   
23   
24   
25   
26   
27
jnp.exp Expand_dims(output_layer(z), axis=(1)), [1, z.shape[1], 1, 1]
Listing 18: The triplet representation of the 8th seed model, which simply applies a linear layer and tiles the output to maintain dimensional consistency.
def get triplet msgs_v9(z, edge_fts, graph_fts, nb_triplet_fts, out_size):  
    def rep_fn(x, size):  
        proj = hk.nets.MLP(size)  
        ff = hk.nets.MLP(size * 8, size)]  
        return proj(x) * ff(x)  
    triplet_node_reps = [rep_fn(z, nb_triplet_fts) for _ in range (3)]  
    triplet_edge_reps = [rep_fn(edge_fts, nb Triplet_fts) for _ in range (3)]  
    triplet_graph_rep = rep_fn(graph_fts, nb Triplet_fts)  
    node_pair.permutations = [(2, 3), (1, 3), (1, 2)]  
    triplets = functools.reduce( lambda x, y: x + y, [ jnp.expand_dims(tri_node_rep, axis=perm) for tri_node_rep, perm in zip( triplet_node_reps, node_pair.permutations ] ],  
    triplets += functools.reduce( lambda x, y: x + y, [ jnp.expand_dims(tri_edge_rep, axis=i) for tri_edge_rep, i in zip(triplet_edge_reps, range(3, 0, -1)) ],  
    triplets += jnp.expand_dims(triplet_graph_rep, axis=(1, 2, 3))  
    return rep_fn(jnp.max(triplets, axis=1), out_size)
Listing 19: The triplet representation of the 9th seed model, which uses a bilinear representation for the node, edge, and graph representations.
A.6 OOD Evaluation of Newly Discovered Models on CLRS
Table 3: A full list comparing 5 of the newly discovered GNNs against the baseline Triplet-GMPNN model from Ibarz et al. (2022) on all 30 of the CLRS (Veličković et al., 2022) tasks. For the baseline, we include the OOD accuracy of both our implementation of the Triplet-GMPNN as well as the number listed in the original paper.
Algorithm	Best Performing Model	Model Size ↓	OOD Accuracy ↑
Best Per-forming Model	Baseline model	Best per-forming newly discovered model	Baseline model (our implementation)	Baseline model (from Ibarz et al. (2022))
Activity Selector	Baseline	262204	262204	95.05 ± 0.53%	93.96± 0.29%	95.18± 0.45%
Articulation Points	QUADNODEM-INMAX	497969	531913	93.46 ± 1.77%	91.40± 1.74%	88.32± 2.01%
Bellman Ford	CONCATREP	568660	524604	97.50 ± 0.31%	97.08± 0.24%	97.39± 0.19%
BFS	MAXMEAN	522931	523963	99.99 ± 0.01%	99.80± 0.04%	99.73± 0.04%
Binary Search	Baseline	262204	262204	77.98 ± 2.49%	79.57± 1.73%	77.58± 2.35%
Bridges	CONCATREP	576612	532556	97.57 ± 1.08%	97.31± 1.11%	93.99± 2.07%
Bubble Sort	CONCATREP	568533	524477	88.87 ± 2.77%	83.20± 4.27%	67.68± 5.50%
DAG Shortest Paths	Baseline	793287	793287	98.01 ± 0.22%	97.48± 0.37%	98.19± 0.30%
DFS	DIV2MEAN	660158	661190	68.14 ± 1.38%	46.78± 3.85%	47.79± 4.19%
Dijkstra	DIV2MEAN	524854	525886	97.30 ± 0.28%	95.94± 0.66%	96.05± 0.60%
Find Maximum Subarray Kadane	Baseline	261290	264514	75.35 ± 0.92%	74.09± 0.83%	76.36± 0.43%
Floyd Warshall	CONCATREP	669145	625089	61.43 ± 0.79%	48.95± 0.49%	48.52± 1.04%
Graham Scan	MAXMEAN	397377	398409	93.76 ± 0.85%	92.72± 2.38%	93.62± 0.91%
Heapsort	CONCATREP	703710	659654	69.90 ± 4.17%	19.45± 5.35%	31.04± 5.82%
Insertion Sort	DIV2MEAN	523445	524477	89.47 ± 2.57%	86.89± 1.89%	78.14± 4.64%
Jarvis March	Baseline	308954	264898	90.36 ± 0.65%	88.91± 0.91%	91.01± 1.30%
Knuth-Morris-Pratt	Baseline	396989	398021	16.29 ± 4.36%	8.88± 1.76%	19.51± 4.57%
LCS Length	Baseline	270419	270419	85.75 ± 0.80%	86.05± 0.65%	80.51± 1.84%
Matrix Chain Order	Baseline	624448	624448	90.77 ± 0.75%	91.15± 0.85%	91.68± 0.59%
Minimum	DIV2MEAN	260275	261307	98.40 ± 0.16%	98.26± 0.26%	97.78± 0.55%
Algorithm	Best Performing Model	Continuation of Table 3
Model Size ↓	OOD Accuracy ↑
Best Per-form-ing Model	Baseline model	Best per-forming newly discovered model	Baseline model (our implementation)	Baseline model (from Ibarz et al. (2022))
MST Kruskal	CONCATREP	443747	399691	91.47 ± 0.48%	90.60± 0.32%	89.80± 0.77%
MST Prim	CONCATREP	569942	525886	88.74 ± 1.67%	85.18± 2.24%	86.39± 1.33%
Naive String Matcher	QUADNODEMINMAX	259364	262588	79.77 ± 2.88%	73.39± 6.33%	78.67± 4.99%
Optimal BST	DIV2MEAN	624955	625987	78.66 ± 0.46%	78.08± 0.96%	73.77± 1.48%
Quickslect	QUADNODEMINMAX	377130	395714	0.79 ± 0.41%	0.13± 0.08%	0.47± 0.25%
Quicksort	DIV2MEAN	524727	525759	85.23 ± 4.26%	84.71± 2.66%	64.64± 5.12%
Segments Intersect	DIV2MEAN	262327	263359	98.15 ± 0.00%	97.40± 0.00%	97.64± 0.09%
Strongly Connected Components	Baseline	707299	663243	41.86 ± 3.39%	43.71± 5.94%	43.43± 3.15%
Task Scheduling Topological Sort	TANHEX-PANDTRIPLETS PANDTRIPLETS	262333	262333	88.23 ± 0.44%	88.10± 0.31%	87.25± 0.35%
88.12 ± 4.71%	76.88± 5.05%	87.27± 2.67%
End of Table 3
A.7 NATS-Bench
We make a brief comparison of EVOPROMPTING against other NAS algorithms on the NATS-Bench Size Search Space (Dong et al., 2021). However, we note that this comparison is limited because it handicaps EVOPROMPTING in multiple ways:
1. Our LLM was pre-trained to generate code, but for this comparison it is prompted to generate the NATS-Bench style architecture strings, which are in the format "64:64:64:64:64," which removes the benefit of its code pre-training.
2. One of EVOPROMPTING main advantages is its open-ended search space. Since its search space does not need to be hand-designed, EVOPROMPTING can potentially discover novel architectures that other NAS algorithms cannot.
Table 4: EVOPROMPTING versus other standard neural architecture search algorithms on the NATSBench size search space. Since properly tuning the hyperparameter values for all search techniques is non-trivial, we obtain the accuracy numbers for all methods other than ours from Dong et al. (2021). EVOPROMPTING performs competitively compared to all the other techniques.
Method	CIFAR-10	CIFAR-100	ImageNet-16-120
Val	Test	Val	Test	Val	Test
Multi-trial	REA	90.37±0.20	93.22±0.16	70.23±0.50	70.11±0.61	45.30±0.69	45.94±0.92
REINFORCE	90.25±0.23	93.16±0.21	69.84±0.59	69.96±0.57	45.06±0.77	45.71±0.93
RANDOM	90.10±0.26	93.03±0.25	69.57±0.57	69.72±0.61	45.01±0.74	45.42±0.86
BOHB	90.07±0.28	93.01±0.24	69.75±0.60	69.90±0.60	45.11±0.69	45.56±0.81
Weight-sharing	channel-wise interpolation	90.71±0.00	93.40±0.00	70.30±0.00	70.72±0.00	44.73±0.00	47.17±0.00
masking + Gumbel-Softmax	90.41±0.10	93.14±0.13	70.30±0.00	70.72±0.00	45.71±0.39	46.38±0.27
masking + sampling	89.73±0.37	92.78±0.30	69.67±0.22	70.11±0.33	44.70±0.60	45.11±0.76
EVOPROMPT-ING	90.38±0.33	93.11±0.90	70.47±0.23	70.39±0.58	45.32±0.26	45.15±0.51
A.8 Broader Impacts
Our work may have a number of ethical, societal, and other broader impacts. Since we focus on automatic improvement of large language models, the implications of our research are largely similar to those of LMs in general. On the one hand, improving the abilities and decreasing the sizes of LMs may increase their accessibility (Köpf et al., 2023), improve energy efficiency (McDonald et al., 2022; Chen et al., 2023), and expand educational and professional opportunities (Kasneci et al., 2023; Eysenbach, 2023). On the other, LMs have long been known to give rise to unjust and toxic language that may hurt and amplify stereotypes (Nadeem et al., 2020; Lucy & Bamman, 2021), exclusionary norms, and allocational harms to marginalized groups (Bender et al., 2021). LMs may also present information hazards, often generating realistic-sounding misinformation (Bickmore et al., 2018; Quach, 2022) or revealing private personal information (Carlini et al., 2021). Lastly, other harms may arise from the ways that humans interact with LMs – either by inadvertently relying too much on unreliable LM outputs (McKee et al., 2021) or via malicious uses (Ranade et al., 2021; Boiko et al., 2023).
Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, pp. 610-623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383097. doi: 10.1145/3442188.3445922. URL https://doi.org/10.1145/3442188.3445922.
Bickmore, T. W., Trinh, H., Olafsson, S., O'Leary, T. K., Asadi, R., Rickles, N. M., and Cruz, R. Patient and consumer safety risks when using conversational assistants for medical information: An observational study of siri, alexa, and google assistant. J Med Internet Res, 20(9), Sep 2018. ISSN 1438-8871. doi: 10.2196/11510. URL http://www.jmir.org/2018/9/e11510/.
Boiko, D. A., MacKnight, R., and Gomes, G. Emergent autonomous scientific research capabilities of large language models, 2023.
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., Oprea, A., and Raffel, C. Extracting training data from large language models. In USENIX Security Symposium, 2021. URL https://arxiv.org/abs/2012.07805.
Chen, L., Zaharia, M., and Zou, J. Frugalgpt: How to use large language models while reducing cost and improving performance, 2023.
Eysenbach, G. The role of chatgpt, generative language models, and artificial intelligence in medical education: A conversation with chatgpt and a call for papers. JMIR Med Educ, 9:e46885, Mar 2023. ISSN 2369-3762. doi: 10.2196/46885. URL https://mededu.jmir.org/2023/1/e46885.
Kasneci, E., Sessler, K., Kuchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Gunnemann, S., Hüllermeier, E., Krusche, S., Kutniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., Stadler, M., Weller, J., Kuhn, J., and Kasneci, G. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and Individual Differences, 103:102274, 2023. doi: https://doi.org/10.1016/j.lindif.2023.102274.
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z.-R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., ES, S., Suri, S., Glushkov, D., Dantuluri, A., Maguire, A., Schuhmann, C., Nguyen, H., and Mattick, A. Openassistant conversations – democratizing large language model alignment, 2023.
Lucy, L. and Bamman, D. Gender and representation bias in GPT-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding, pp. 48-55, Virtual, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.nuse-1.5. URL https://aclanthology.org/2021.nuse-1.5.
McDonald, J., Li, B., Frey, N., Tiwari, D., Gadepally, V., and Samsi, S. Great power, great responsibility: Recommendations for reducing energy for training language models. In Findings of the Association for Computational Linguistics: NAACL 2022. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022-findings-naacl.151. URL https://doi.org/10.18653%2Fv1%2F2022-findings-naacl.151.
McKee, K. R., Bai, X., and Fiske, S. Humans perceive warmth and competence in artificial intelligence, Feb 2021. URL psyxarxiv.com/5ursp.
Nadeem, M., Bethke, A., and Reddy, S. Stereoset: Measuring stereotypical bias in pretrained language models, 2020.
Quach, K. Researchers made an openai gpt-3 medical chatbot as an experiment. it told a mock patient to kill themselves, Aug 2022. URL https://www.theregister.com/2020/10/28/gpt3_medical-chatbot_expperiment/.
Ranade, P., Piplai, A., Mittal, S., Joshi, A., and Finin, T. Generating fake cyber threat intelligence using transformer-based models, 2021.