- Task 4: Shortest Path The shortest path between two nodes is the path with the sum of edge weights minimized. Given an undirected graph  $\mathcal{G} = \{\mathcal{V},\mathcal{E}\}$ , a positive weight  $w$  for each edge, and two nodes  $u$  and  $v$ , the task is to find the shortest path between node  $u$  and node  $v$  and its corresponding path length. We filter the generated base graphs by specifying that the number of nodes on the correct shortest path is as least  $\ell$ , where  $\ell$  is chosen from  $\ell_{\min}$  to  $\ell_{\max}$  for each question to sever as an additional difficulty control measure. We adopt two metrics: exact match, where the LLM solution is a valid path and optimal, and partial credit, where the LLM solution is valid and is the rank- $i$ -th shortest among all the possible paths (i.e. the number of shorter paths plus one). The partial credit score for each problem can be formulated as  $\mathrm{PC} = 1 / \mathrm{rank}_i$ .
- Task 5: Maximum Flow Let  $\mathcal{G} = \{\mathcal{V}, \mathcal{E}\}$  be a network with two nodes  $s, t \in \mathcal{V}$  being the source and the sink. Each edge is associated with a capacity  $c$ , the maximum amount of flow that can pass through the edge. We ask LLMs to generate a plan to route as much flow as possible from the source to the sink. We evaluate in both exact match with the optimal plan and partial credit for this task, where partial credit can be formulated as  $\mathrm{PC} = \begin{cases} t / s, & \text{if } t \leq s \\ 0, & \text{if } t > s \end{cases}$ , where  $s$  is the flow value under the optimal plan, and  $t$  is the flow value of the solution generated by LLMs.
- Task 6: Bipartite Graph Matching In an undirected graph  $\mathcal{G} = \{\mathcal{V}, \mathcal{E}\}$ , a matching is a set of edges without common nodes. A bipartite graph is a graph whose nodes can be divided into two disjoint sets  $\mathbf{U}$  and  $\mathbf{V}$ , and in each set no nodes are adjacent to each other. Given a bipartite graph, the task is to find the matching that maximizes the number of edges. We use an external program to evaluate whether the solution generated by LLMs is valid and optimal.
- Task 7: Hamilton Path In an undirected graph, a Hamilton path is a path that visits every node exactly once. Given an undirected graph  $\mathcal{G} = \{\mathcal{V}, \mathcal{E}\}$ , the task is to find a valid Hamilton path. We filter generated base graphs to ensure that at least one valid Hamilton path exists and use an external program to evaluate the LLM solution.
- Task 8: Graph Neural Networks Given an undirected graph  $\mathcal{G} = \{\mathcal{V}, \mathcal{E}\}$ , and a two-dimension node embedding  $\mathbf{x}_i$  for each node, the task is to perform  $\ell$  layers of message passing, i.e. to update the node embedding with the sum of all the neighbors' embeddings. Formally,  $\mathbf{x}_i^{(\ell + 1)} = \sum_{j \in \mathcal{N}_i} \mathbf{x}_j^{(\ell)}$  where  $\mathcal{N}_i$  denotes the neighborhood of node  $i$  and  $(\ell)$  denotes the  $\ell$ -th layer. We use an exact match with the correct node embeddings and two types of partial credits for this task. Specifically, the first partial credit is the percentage of the nodes whose embedding is correct (PC), and the second is the average of all the values' relative errors for the standard answer (RE). The relative error is formulated as  $R_E = \frac{|x - y|}{\max(x, y)}$ , where  $x$  is the value generated by LLMs and  $y$  is the value in the standard answer, averaged across all embedding dimensions.