SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure A.23: Train Problem 3090, Solution 9
1 find_2nd-largest(arr): find_2nd-largest takes a list of numbers and returns the second  $\leftrightarrow$  largest number in the list.   
2 filter_int(arr): filter_int takes an array of integers and returns an array of integers.   
3 sec-big(a, b): sec-big takes two numbers and returns the smaller of the two.   
4 sort(arr): sort takes an array of numbers and returns a sorted array of numbers.   
5 is_diff(arr): is_diff takes an array of numbers and returns True if there are any two  $\leftrightarrow$  numbers in the array that are different, and False if all the numbers in the array are  $\leftrightarrow$  the same.
Figure A.24: Train Problem 3090, Solution 9
1 # find_2nd-largest takes a list of numbers and returns the second largest number in the list.  
2 def find_2nd-largest(arr):  
3 if len(arr) == 1:  
4 return None  
5 arr = filter_int(arr)  
6 if len(arr) == 0:  
7 return None  
8 if len(arr) == 1:  
9 return arr[0]  
10 arr = sort(arr)  
11 if is_diff(arr) == False:  
12 return None  
13 res = arr[len(arr) - 2]  
14 return res  
15  
16 # filter_int takes an array of integers and returns an array of integers.  
17 def filter_int(arr):  
18 return list.filter(lambda x: type(x) == int, arr))  
19  
20 # sec-big takes two numbers and returns the smaller of the two.  
21 def sec/big(a, b):  
22 if a > b:  
23 return b  
24 else:  
25 return a  
26  
27 # sort takes an array of numbers and returns a sorted array of numbers.  
28 def sort(arr):  
29 return sorted(arr)  
30  
31 # is_diff takes an array of numbers and returns True if there are any two numbers in the  $\hookrightarrow$  array that are different, and False if all the numbers in the array are the same.  
32 def is_diff(arr):  
33 for i in range(len(arr) - 1):  
34 if arr[i] != arr[i + 1]:  
35 return True  
36 return False
Figure A.25: Train Problem 3140, Solution 7
1 happyNumbers(n): happyNumbers takes a positive integer n and returns a list of all the  $\leftrightarrow$  happy numbers between 1 and n, inclusive.   
2 is_happy_number(number): _is_happy_number takes a positive integer and returns True if the  $\leftrightarrow$  number is a happy number, False otherwise.   
3 sum_squares(number): _sum_squares takes a non-negative integer and returns the sum of  $\leftrightarrow$  the squares of its digits.
1 # happyNumbers takes a positive integer n and returns a list of all the happy numbers  $\rightarrow$  between 1 and n, inclusive.   
2 def happyNumbers(n): return [i for i in range(1, n + 1) if _is_happy_number(i)]   
3   
4   
5 # is happy number takes a positive integer and returns True if the number is a happy number,  $\leftrightarrow$  False otherwise.   
6 def _is_happy_number(number): # We want to make sure that the number is positive if number  $<  0$  .. return False   
9   
10 # We want to make sure that the number is not 1 if number  $= = 1$  .. return True   
12   
13 # We want to keep track of the numbers we have used used(numbers  $= []$    
15 # We want to loop through the number while number not in used(numbers: # We want to add the number to the list of used numbers used(numbers.append(number) # We want to find the sum of the squares of the digits of the number number  $=$  _sum_squares(number) # We want to check if the number is one if number  $= = 1$  . # We want to return True return True   
23   
24   
25 # We want to return False   
26 return False   
27   
28 # _sum_squares takes a non-negative integer and returns the sum of the squares of its digits.   
29 def _sum_squares(number): if number  $<  0$  .. raise ValueError else: number  $=$  str(number) sum  $= 0$  for i in number: sum  $+ =$  int(i) \*\* 2 return sum
Figure A.26: Train Problem 3140, Solution 7
1 am_i_wilson(n): am_i_wilson(n) returns True if n is a prime number between 2 and 563,  $\hookrightarrow$  inclusive, and False otherwise.  
2 is_prime(n): is_prime takes a positive integer n and returns True if n is prime and False  $\hookrightarrow$  otherwise.  
3 factorial(n): factorial(n) returns the product of all integers from 1 to n, inclusive.
Figure A.27: Train Problem 3229, Solution 26
1 # am_i_wilson(n) returns True if n is a prime number between 2 and 563, inclusive, and False  $\rightarrow$  otherwise.   
2 def am_i_wilson(n):   
3 if is_prime(n) and  $2\leqslant n$  and  $n <   = 563$  .. return (factorial(n-1) + 1) % (n**2)  $= = 0$  else:   
6 return False   
7   
8 # is_prime takes a positive integer n and returns True if n is prime and False otherwise.   
9 def is_prime(n):   
10 if n  $= = 2$  . return True   
11 if n  $= = 3$  .. return True   
13 if n % 2  $= = 0$  .. return False   
14 if n % 3  $= = 0$  .. return False   
17 i  $= 5$    
19 w  $= 2$    
20 while i \* i <= n: if n % i  $= = 0$  .. return False   
22 i  $+ = w$    
24 w  $= 6 - w$    
25 return True   
26   
27   
28 # factorial(n) returns the product of all integers from 1 to n, inclusive.   
def factorial(n):   
30 if n  $= = 0$  .. return 1   
31 else: return n \* factorial(n-1)
Figure A.28: Train Problem 3229, Solution 26
1 am_i_wilson(n): am_i_wilson takes a positive integer  $n$  and returns True if  $n$  is prime and (n  $\hookrightarrow$  -1)! + 1 is divisible by  $n^2$ , and False otherwise.
2 fac(n): fac is a function that takes a positive integer  $n$  and returns the product of all integers from 1 to  $n$ .
3 is_prime(n): is_prime takes a positive integer  $n$  and returns True if  $n$  is prime and False otherwise.
Figure A.29: Train Problem 3229, Solution 71
1 # am_i_wilson takes a positive integer n and returns True if n is prime and (n-1)! + 1 is  $\rightarrow$  divisible by  $n^{\prime}2$  , and False otherwise.   
2 def am_i_wilson(n): return is_prime(n) and (fac(n-1) + 1) % n**2 == 0   
4   
5 # fac is a function that takes a positive integer n and returns the product of all integers  $\rightarrow$  from 1 to n.   
6 def fac(n): if n  $= = 0$  . return 1   
9 return n * fac(n-1)   
10 # is_prime takes a positive integer n and returns True if n is prime and False otherwise.   
11 def is_prime(n): if n  $= = 2$  return True elif n < 2 or n % 2  $= = 0$  .. return False for i in range(3, int(n**0.5)+1, 2): if n % i  $= = 0$  .. return False   
18   
19 return True
Figure A.30: Train Problem 3229, Solution 71
1 evil(n): evil(n) returns "It's Evil!" if n is an evil number, otherwise it returns "It's  $\hookrightarrow$  Odious!" The range of evil is the set of all integers, and the domain is the set of all strings.  
2 evilometer(n): evilometer(n) is a generator that yields n times if n is even, and yields n  $\hookrightarrow$  // 2 times if n is odd.
Figure A.31: Train Problem 3321, Solution 33
1 # evil(n) returns "It's Evil!" if n is an evil number, otherwise it returns "It's Odious!"
 $\hookrightarrow$  The range of evil is the set of all integers, and the domain is the set of all strings.
2 def evil(n):
3 gen = vilometer(n)
4 if sum(list(gen)) % 2 == 0:
5 return "It'sOdious!"
6 else:
7 return "It'sOdious!"
8
9 # vilometer(n) is a generator that yields n times if n is even, and yields n // 2 times if n  $\hookrightarrow$  is odd.
10 def vilometer(n):
11 while n:
12 yield n
13 if n % 2:
14 n //= 2
15 else:
16 n -= 1
Figure A.32: Train Problem 3321, Solution 33
1 circular_prime(number): circular_prime takes a number and returns True if it is a circular  $\hookrightarrow$  prime, and False otherwise.  
2 rotate(1, n): rotate takes a list and an integer and returns a new list with the last n  $\hookrightarrow$  elements of the original list moved to the front.  
3 is_prime(n): is_prime returns True if n is a prime number, and False otherwise.
Figure A.33: Train Problem 3368, Solution 5
Figure A.34: Train Problem 3368, Solution 5
1 # circular_prime takes a number and returns True if it is a circular prime, and False  $\leftrightarrow$  otherwise.   
2 def circular_prime(number): if is_prime(number)  $= =$  False: return False else: for i in range(len(str(number))-1): number  $=$  rotate(str(number), 1) if is_prime(int(number))  $= =$  False: return False return True   
12 # rotate takes a list and an integer and returns a new list with the last n elements of the  $\leftrightarrow$  original list moved to the front.   
13 def rotate(1, n): return 1[-n:] + 1[: -n]   
15   
16 # is_prime returns True if n is a prime number, and False otherwise.   
17 def is_prime(n): if n < 2: return False for i in range(2, int(n  $^ { \text{串} } \times 0 . 5 ) + 1$  .. if n % i  $= = 0$  : return False   
22
Figure A.35: Train Problem 3368, Solution 6
1 circular_prime(n): circular_prime(n) returns True if n is a circular prime, and False  $\hookrightarrow$  otherwise.   
2 is_prime(n): is_prime takes a number n and returns True if n is prime and False if n is not  $\hookrightarrow$  prime.   
3 circul_num(n, l): circul_num takes an integer n and an integer 1, and returns the integer  $\hookrightarrow$  that results from moving the first digit of n to the end of n, and then padding the  $\hookrightarrow$  result with zeros until it has length 1.
1 # circular_prime(n) returns True if n is a circular prime, and False otherwise.   
2 def circular_prime(n):   
3 # Check if n is a prime number.   
4 if not is_prime(n):   
5 return False   
6 else:   
7 # Get the length of n.   
8 length  $=$  len(str(n))   
9 # Get the first digit of n.   
10 firstdigit  $\equiv$  str(n)[0]   
11 # Get the number of rotations.   
12 rotations  $=$  length   
13 # Rotate n.   
14 while rotations  $>0$  ..   
15 # Get the new number..   
16 n  $=$  circul_num(n, length)   
17 # Check if n is a prime number.   
18 if not is_prime(n):   
19 return False   
20 # Decrement the number of rotations.   
21 rotations  $- = 1$    
22 return True   
23 # is_prime takes a number n and returns True if n is prime and False if n is not prime.   
24 def is_prime(n):   
25 if n  $= = 2$  or n  $= = 3$  ..   
26 return True   
27 if n % 2  $= = 0$  or n  $<  2$  ..   
28 return False   
29 for i in range(3, int(n\*\*0.5)+1, 2): # only odd numbers   
30 if n % i  $= = 0$  ..   
31 return False   
32 return True   
33   
34 # circul_num takes an integer n and an integer l, and returns the integer that results from  $\hookrightarrow$  moving the first digit of n to the end of n, and then padding the result with zeros  $\hookrightarrow$  until it has length 1.   
35 def circul_num(n, 1):   
36 n  $=$  str(n)   
37 digit  $= \mathrm{n}[0]$    
38 n  $= \mathrm{n}[1:]$  + digit   
39 n  $= \mathrm{n.ljust}(1,$  "O")   
40 return int(n)
Figure A.36: Train Problem 3368, Solution 6
1 list_squared(m, n): list_squared(m, n) returns a list of lists of integers, where each  $\hookrightarrow$  sublist contains an integer and its sum of squared divisors, and the integers are in the range [m, n] (inclusive).  
2 divisors_list(num): divisors_list takes a number and returns a list of all the divisors of  $\hookrightarrow$  that number.  
3 sum_squares(nums): sum_squares takes a list of numbers and returns the sum of the squares  $\hookrightarrow$  of those numbers.  
4 isSquarable(num): isSquarable takes a number and returns True if the number is a perfect square, and False otherwise.
Figure A.37: Train Problem 3393, Solution 35
1 # list_squared(m, n) returns a list of lists of integers, where each sublist contains an  $\leftrightarrow$  integer and its sum of squared divisors, and the integers are in the range [m, n] (  $\leftrightarrow$  inclusive).   
2 def list_squared(m, n): result  $=$  [] for num in range(m, n + 1): divisors  $=$  divisors_list(num) sum_of_squares  $=$  sum_squares(divisors) if isSquarable(sum_of_squares): result.append([num, sum_of_squares]) return result   
10 # divisors_list takes a number and returns a list of all the divisors of that number.   
11 def divisors_list(num): divisors  $=$  [] for i in range(1, num + 1): if num % i  $= = 0$  : divisors.append(i) return divisors   
17   
18   
19 # sum_squares takes a list of numbers and returns the sum of the squares of those numbers.   
20 def sum_squares(nums): sum  $= 0$  for i in range(len(nums)): sum  $+ =$  nums[i]\*\*2 return sum   
24   
25   
26 # isSquarable takes a number and returns True if the number is a perfect square, and False  $\leftrightarrow$  otherwise.   
27 def isSquarable(num): root  $=$  num \*\* (1/2) if root % 1  $= = 0$  : return True else: return False
Figure A.38: Train Problem 3393, Solution 35
M Compiled Examples
1 # Takes a board and returns the next iteration of the game of life, but with all values  $\rightarrow$  flipped   
2 def game_of-Life_inversion_iteration(array_at_time_t):   
3 # Your code here   
4 #return game_of-Life_iteration(invert_array(array_at_time_t))   
5 return invert_array(game_of-Life_iteration(array_at_time_t))   
6   
7 # Takes a board and returns the next iteration of the game of life   
8 def invert_array(array_at_time_t):   
9 return [list(map( lambda x: 1-x, row)) for row in array_at_time_t]   
10   
11 # Takes a board and returns the board with all values flipped   
12 def game_of-Life_iteration(array_at_time_t):   
13 # The array that will be returned   
14 array_at_time_t_plus_1 = []   
15   
16 # Iterate through the rows of the array   
17 for i in range(0, len(array_at_time_t)):   
18 # The array that will contain the next row   
19 next_row = []   
20   
21 # Iterate through the columns of the array   
22 for j in range(0, len(array_at_time_t[i]):   
23 # The number of neighbors   
24 num_neighbors = 0   
25   
26 # Iterate through the neighbors of the cell   
27 for k in range(-1, 2):   
28 for l in range(-1, 2):   
29 # Don't count the cell itself   
30 if k == 0 and l == 0: continue   
31   
32   
33 # Check if the neighbor is valid   
34 if i + k >= 0 and i + k < len(array_at_time_t) and j + l >= 0 and j + l <  $\rightarrow$  len(array_at_time_t[i]):   
35 # If the neighbor is alive, increment the number of neighbors   
36 if array_at_time_t[i + k][j + l] == 1: num_neighbors += 1   
37   
38   
39 # If the cell is alive, check if it should die   
40 if array_at_time_t[i][j] == 1: if num_neighbors < 2 or num_neighbors > 3: next_row.append(0) else: next_row.append(1)   
41   
42   
43   
44   
45 # If the cell is dead, check if it should become alive else: if num_neighbors == 3: next_row.append(1) else: next_row.append(0)   
46   
47 # Add the next row to the array   
48 array_at_time_t_plus_1.append(next_row)   
49   
50   
51   
52   
53 assert game_of-Life_inversion_iteration([[0, 0, 1], [1, 0, 0], [1, 0, 0]]) == [[1, 1, 1], [1,  $\rightarrow$  0, 1], [1, 1, 1]]   
54 assert game_of-Life_inversion_iteration([[0, 1, 0, 0], [1, 0, 1, 0], [1, 0, 0, 1], [0, 1, 1,  $\rightarrow$  0]) == [[1, 0, 1, 1], [0, 1, 0, 1], [0, 1, 1, 0], [1, 0, 0, 1]]
Figure A.39: Compilation of the program taking Conway's game of life [24] and returning the next state with the values inverted. In this case, we use automatic decomposition to generate the two
subfunctions of game_of_life_iteration.
1 Takes a board with active and inactive cells and returns the next iteration of the game of  $\rightarrow$  life, but with all the values flipped   
2 def game_of-Life_inversion_iteration(array_at_time_t):   
3 return array_inversion(game_of-Life_iteration(array_at_time_t))   
4   
5 # Takes a board with active and inactive cells as a list of lists and returns the next  $\rightarrow$  iteration of the game of life   
6 def game_of-Life_iteration(array_at_time_t):   
7 array_at_time_t_plus_1 = []   
8 for row in range(len(array_at_time_t):   
9 array_at_time_t_plus_1.append([])   
10 for column in range(len(array_at_time_t[row]):   
array_at_time_t_plus_1[row].append(0)   
11 for row in range(len(array_at_time_t)):   
13 for column in range(len(array_at_time_t[row]):   
14 if array_at_time_t[row][column]  $= = 1$  .. if count_neighbors(array_at_time_t, row, column)  $<  2$  .. array_at_time_t_plus_1[row][column]  $= 0$    
16 else:   
20 array at time t_plus_1[row][column]  $= 1$    
21 else:   
22 if count_neighbors(array at time t, row, column)  $= = 3$  .. array at time t_plus_1[row][column]  $= 1$    
23 return array at time t_plus_1   
24 return array at time t plus 1   
25   
26 # Invert a square array by replacing all O's with 1's and vice versa   
27 def array_inversion(array):   
28 inverted_array  $= []$    
29 for i in range(len(array)):   
30 inverted_array.append([])   
31 for j in range(len(array[i]): inverted_array[i].append(1 - array[i][j])   
32 return inverted_array   
33   
34   
35 #   
36 def count_neighbors(array at time_t, row, column):   
count  $= 0$    
37 for i in range(row-1, row+2):   
40 for j in range(column-1, column+2): if i  $= =$  row and j  $= =$  column: continue if i < 0 or j < 0: continue if i >= len(array at time t) or j >= len(array at time t[0]): continue if array at time t[i][j]  $= = 1$  .. count += 1   
return count   
50 assert game_of-Life_inversion_iteration([[0, 0, 1], [1, 0, 0], [1, 0, 0]])  $= = [1, 1, 1], [1,$ $\leftrightarrow 0, 1], [1, 1, 1]$    
52 assert game_of-Life_inversion_iteration([[0, 1, 0, 0], [1, 0, 1, 0], [1, 0, 0, 1], [0, 1, 1,  $\leftrightarrow 0]] = = [1, 0, 1, 1], [0, 1, 0, 1], [1, 0, 0, 0, 1]$    
53 assert game_of-Life_iteration([[0, 0, 1], [1, 0, 0], [1, 0, 0]])  $= = [0, 0, 0], [0, 1, 0], [0,$ $\leftrightarrow 0, 0]$    
55 assert game_of-Life_iteration([[0, 1, 0, 0], [1, 0, 1, 0], [1, 0, 0, 1], [0, 1, 1, 0]])  $= =$ $\leftrightarrow [0, 1, 0, 0], [1, 0, 1, 0], [1, 0, 0, 1], [0, 1, 1, 0]$    
56 assert array_inversion([1]])  $= = [0]$    
58 assert array_inversion([0, 1], [1, 0]])  $= = [1, 0], [0, 1]$
Figure A.40: Compilation of the program taking Conway's game of life [24] and returning the next state, with the values inverted. In this case, we use automatic infilling to generate the count_neighbors function.
1 # Calls base_case if 1, otherwise recursion_rule
2 def collatz_recursion(num, cur_list=list():
3     if num == 1:
4         return base_CASE(num, cur_list)
5         else:
6         return recursion_rule(num, cur_list)
7
8 # Returns the list with the number appended to it
9 def base(case(num, cur_list):
10         cur_list.append(num)
11         return cur_list
12
13
14 # Add num to list, collatz with  $3n + 1$  if odd or n / 2 if even
15 def recursion_rule(num, cur_list):
16         cur_list.append(num)
17         if num % 2 == 0:
18         return collatz_recursion(num / 2, cur_list)
19         else:
20         return collatz_recursion((3 * num) + 1, cur_list)
21
22
23 assert collatz_recursion(19) == [19, 58, 29, 88, 44, 22, 11, 34, 17, 52, 26, 13, 40, 20, 10,
 $\leftrightarrow$  5, 16, 8, 4, 2, 1]
Figure A.41: Compilation of the program generating a list corresponding to the Collatz conjecture.
1 -- if p  $\wedge$  q, then q  $\wedge$  p   
2 lemma p_q_implies_q_p(p q: Prop):   
3  $p\wedge q\to q\wedge p:=$    
4 begin   
6 intro h,   
7 cases h with hp hq,   
8 split,   
9 exact hq,   
10 exact hp,   
11 end   
12   
13 -- Description: if p  $\vee$  q, then q  $\vee$  p   
14 -- if q  $\wedge$  p, then p  $\wedge$  q   
15 lemma q_p_implies_p_q(p q: Prop):   
16  $(q\wedge p)\rightarrow (p\wedge q)\coloneqq$    
18 begin   
19 intro h,   
20 split,   
21 exact h.right,   
22 exact h.left,   
23 end   
24   
25 /-   
26 Theorem:   
27 If q  $\wedge$  p, then p  $\wedge$  q   
28 -/   
29 -- the and operator is commutative   
30 lemma and_commute(p q: Prop):   
31  $(p\wedge q\to q\wedge p)\wedge (q\wedge p\to p\wedge q)\coloneqq$    
32   
33 begin   
34 apply and.intro,   
35 { apply p_q_implies_q_p },   
36 { apply q_p_implies_p_q }   
37 end   
38   
39 -- Description: if p  $\wedge$  q, then p   
40 -- Signature: p_and_q_implies_p(p q: Prop)   
41   
42 -- show  $(p\wedge q\to q\wedge p)\wedge (q\wedge p\to p\wedge q)$
Figure A.42: Generated proof of and_commute. We trim the post-proof comments elsewhere.
Figure A.43: Generated solution for select_Airport_cities.
1 # chess_game takes a board and returns the winner of the game.   
2 def chess_game(board):   
3 if weight(board) \(= = 0\) . return 'Draw'   
4 if weight(board) \(>0\) .. return 'White'   
5 if weight(board) \(<  0\) .. return 'Black'   
6 def weight(board) : return weight   
7 return weight   
8 def weightboard, row, col, weight):   
9 # weighttakes a board and returns the weight of the board.   
11 def weight(board): weight \(= 0\) for row in range(len(board)): for col in range(len(board[row])): if board[row][col] != '.': weight \(+ =\) weight ,(board, row, col, 0)   
17 return weight   
18   
19 # weight ,(board, row, col, weight):   
21 if board[row][col] \(= = \cdot ,\cdot ,\) return O elif board[row][col] \(= = \cdot Q\) .. return weight + 9 elif board[row][col] \(= = \cdot R\) .. return weight + 5 elif board[row][col] \(= = \cdot B\) .. return weight + 3 elif board[row][col] \(= = \cdot N\) .. return weight + 3 elif board[row][col] \(= = \cdot P\) .. return weight + 1 elif board[row][col] \(= = \cdot q\) .. return weight - 9 elif board[row][col] \(= = \cdot r\) .. return weight - 5 elif board[row][col] \(= = \cdot b\) .. return weight - 3 elif board[row][col] \(= = \cdot n\) .. return weight - 3 elif board[row][col] \(= = \cdot p\) .. return weight - 1 else: return weight   
4 assert repr(str(chess_game('...QK...\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n.. \\(assert repr(str(chess_game('rnpkbnr\npppppplp\nPp.p.n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..
4 assert repr(str(chess_game('rpppppr\nPp.p.n..\\n..\\n..\\n..\\n..\\n..\\n..\\n..
5 assert repr(str(chess_game('...BQ.K\nB...Pp.p.n..\\n..\\n..\\n..
6 return weight
Figure A.44: Generated solution for Problem 368 of the APPS test set, identifying the leader of a chess game from the board.
N APPS Decomposition Prompts and Evaluation Hyperparameters
We slightly loosen the requirements for Parsel programs generated by language models, treating redundant function definitions as references instead of raising errors. We sample everything with temperature  $= 0.6$ , except the translations which we sample with temperature  $= 0.2$ , a presence penalty of 0.1, and a logit bias to prevent it from generating the text "def", as Codex has a tendency to degenerate to producing Python even when prompted with Parsel examples. We allow at most 500 tokens per function, but in practice found that they typically used less than half of them.
For evaluation, we use a timeout of 0.04 seconds per solution and evaluate at most 100,000 implementations per generated Parsel program.
For the Codex-only ablation, we allow it to generate at most 1000 tokens, in large part due to the rate limit. In particular, there is a heuristic rate limit that rejects any calls requesting more than 4,000 tokens. As a result, any larger number of samples per problem would prevent batching more than 3 samples at a time.