Gotcha! global variable with recursion | by Han Qi | Towards Data Science

I was working on this leetcode question https://leetcode.com/contest/weekly-contest-212/problems/path-with-minimum-effort/ using backtracking and spent some time debugging strange output. This article discusses some pitfalls when using recursion with global variables, how to handle them, and how to change the code from global to local.
Disclaimer: Backtracking only passing 15/75 test cases and Time Limit Exceeded for the rest, the purpose of this article is to highlight possible issues with global variables rather than give the best solutions. For more beautiful solutions using Binary Search, Dijkstra and Kruskal, watch Alex’s walkthrough (beginning 7:26) at https://www.youtube.com/watch?v=AM__3Zx1XNw&t=2325s&ab_channel=AlexWice
The problem is to find the path with minimum effort after walking from top left to bottom right with each step allowing 4 directions of up, down, left, right. effort is defined as the maximum absolute difference in values between any 2 consecutive squares.
import numpy as np
import math

min_effort = math.inf

def valid(r,c,visited):
    return 0 <= r < visited.shape[0] and 0 <= c < visited.shape[1] and visited[r,c] == 0

def find_path(heights_arr,visited,r,c,effort,max_row, max_col):
    global min_effort
    
    if r == max_row and c == max_col: 
        print(f'-------------------Found end: effort is {effort}')
        return effort
    
    for row,col in ((r+1,c),(r,c+1),(r-1,c),(r,c-1)):
        if valid(row,col,visited):
            print(f'going from {r},{c} to {row},{col}')
            print(f'effort before recurse: {effort}')
            visited[row,col] = 1 
            updated_effort = max(effort, abs(heights_arr[row,col]-heights_arr[r,c]))  # possibly update current max effort along path
            
            #recursion_res = find_path(heights_arr,visited,row,col,updated_effort,max_row,max_col)
            #print(f'min_effort:{min_effort} recursion_res:{recursion_res}')
            #min_effort = min(min_effort,recursion_res)
            
            min_effort = min(min_effort,find_path(heights_arr,visited,row,col,updated_effort,max_row,max_col))   # problem
            #min_effort = min(find_path(heights_arr,visited,row,col,updated_effort,max_row,max_col),min_effort)
            
            visited[row,col] = 0
            
            print(f'global min_effort after recurse: {min_effort}')
            
    return math.inf
    #return min_effort
    
def minimumEffortPath(heights):
    heights_arr = np.array(heights) 
    rows, cols = heights_arr.shape
    max_row, max_col = rows-1, cols-1
    
    r,c = 0,0
    visited = np.zeros((rows, cols))
    visited[r,c]=1
    
    effort = 0
    find_path(heights_arr,visited,r,c,effort,max_row, max_col)


heights = [[1,2,3],[3,8,4],[5,3,5]]
minimumEffortPath(heights)
print(min_effort)

going from 0,0 to 1,0
effort before recurse: 0
going from 1,0 to 2,0
effort before recurse: 2
going from 2,0 to 2,1
effort before recurse: 2
going from 2,1 to 2,2
effort before recurse: 2
-------------------Found end: effort is 2
global min_effort after recurse: 2
going from 2,1 to 1,1
effort before recurse: 2
going from 1,1 to 1,2
effort before recurse: 5
going from 1,2 to 2,2
effort before recurse: 5
-------------------Found end: effort is 5
global min_effort after recurse: 2
going from 1,2 to 0,2
effort before recurse: 5
going from 0,2 to 0,1
effort before recurse: 5
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
going from 1,1 to 0,1
effort before recurse: 5
going from 0,1 to 0,2
effort before recurse: 6
going from 0,2 to 1,2
effort before recurse: 6
going from 1,2 to 2,2
effort before recurse: 6
-------------------Found end: effort is 6
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: inf
global min_effort after recurse: inf
going from 1,0 to 1,1
effort before recurse: 2
going from 1,1 to 2,1
effort before recurse: 5
going from 2,1 to 2,2
effort before recurse: 5
-------------------Found end: effort is 5
global min_effort after recurse: 5
going from 2,1 to 2,0
effort before recurse: 5
global min_effort after recurse: 5
global min_effort after recurse: inf
going from 1,1 to 1,2
effort before recurse: 5
going from 1,2 to 2,2
effort before recurse: 5
-------------------Found end: effort is 5
global min_effort after recurse: 5
going from 1,2 to 0,2
effort before recurse: 5
going from 0,2 to 0,1
effort before recurse: 5
global min_effort after recurse: 5
global min_effort after recurse: 5
global min_effort after recurse: inf
going from 1,1 to 0,1
effort before recurse: 5
going from 0,1 to 0,2
effort before recurse: 6
going from 0,2 to 1,2
effort before recurse: 6
going from 1,2 to 2,2
effort before recurse: 6
-------------------Found end: effort is 6
global min_effort after recurse: 6
global min_effort after recurse: inf
global min_effort after recurse: inf
global min_effort after recurse: inf
global min_effort after recurse: inf
global min_effort after recurse: inf
going from 0,0 to 0,1
effort before recurse: 0
going from 0,1 to 1,1
effort before recurse: 1
going from 1,1 to 2,1
effort before recurse: 6
going from 2,1 to 2,2
effort before recurse: 6
-------------------Found end: effort is 6
global min_effort after recurse: 6
going from 2,1 to 2,0
effort before recurse: 6
going from 2,0 to 1,0
effort before recurse: 6
global min_effort after recurse: 6
global min_effort after recurse: 6
global min_effort after recurse: inf
going from 1,1 to 1,2
effort before recurse: 6
going from 1,2 to 2,2
effort before recurse: 6
-------------------Found end: effort is 6
global min_effort after recurse: 6
going from 1,2 to 0,2
effort before recurse: 6
global min_effort after recurse: 6
global min_effort after recurse: inf
going from 1,1 to 1,0
effort before recurse: 6
going from 1,0 to 2,0
effort before recurse: 6
going from 2,0 to 2,1
effort before recurse: 6
going from 2,1 to 2,2
effort before recurse: 6
-------------------Found end: effort is 6
global min_effort after recurse: 6
global min_effort after recurse: inf
global min_effort after recurse: inf
global min_effort after recurse: inf
global min_effort after recurse: inf
going from 0,1 to 0,2
effort before recurse: 1
going from 0,2 to 1,2
effort before recurse: 1
going from 1,2 to 2,2
effort before recurse: 1
-------------------Found end: effort is 1
global min_effort after recurse: 1
going from 1,2 to 1,1
effort before recurse: 1
going from 1,1 to 2,1
effort before recurse: 4
going from 2,1 to 2,2
effort before recurse: 5
-------------------Found end: effort is 5
global min_effort after recurse: 1
going from 2,1 to 2,0
effort before recurse: 5
going from 2,0 to 1,0
effort before recurse: 5
global min_effort after recurse: 1
global min_effort after recurse: 1
global min_effort after recurse: 1
going from 1,1 to 1,0
effort before recurse: 4
going from 1,0 to 2,0
effort before recurse: 5
going from 2,0 to 2,1
effort before recurse: 5
going from 2,1 to 2,2
effort before recurse: 5
-------------------Found end: effort is 5
global min_effort after recurse: 1
global min_effort after recurse: 1
global min_effort after recurse: 1
global min_effort after recurse: 1
global min_effort after recurse: 1
global min_effort after recurse: inf
global min_effort after recurse: inf
global min_effort after recurse: inf
inf

import numpy as np
import math

def valid(r,c,visited):
    return 0 <= r < visited.shape[0] and 0 <= c < visited.shape[1] and visited[r,c] == 0

def find_path(heights_arr,visited,r,c,effort,max_row, max_col,min_effort):
    
    if r == max_row and c == max_col: 
        print(f'-------------------Found end: effort is {effort}')
        return effort
    
    for row,col in ((r+1,c),(r,c+1),(r-1,c),(r,c-1)):
        if valid(row,col,visited):
            print(f'going from {r},{c} to {row},{col}')
            print(f'effort before recurse: {effort}')
             
            updated_effort = max(effort, abs(heights_arr[row,col]-heights_arr[r,c]))  # possibly update current max effort along path
            
            visited[row,col] = 1

            #recursion_res = find_path(heights_arr,visited,row,col,updated_effort,max_row,max_col,min_effort)
            #print(f'min_effort:{min_effort} recursion_res:{recursion_res}')
            
            #min_effort = min(min_effort,find_path(heights_arr,visited,row,col,updated_effort,max_row,max_col,min_effort))
            min_effort = min(find_path(heights_arr,visited,row,col,updated_effort,max_row,max_col,min_effort),min_effort)

            visited[row,col] = 0
            print(f'global min_effort after recurse: {min_effort}')
            
    return min_effort
    #return math.inf
    
def minimumEffortPath(heights):
    min_effort = math.inf
    
    heights_arr = np.array(heights) 
    rows, cols = heights_arr.shape
    max_row, max_col = rows-1, cols-1
    
    r,c = 0,0
    visited = np.zeros((rows, cols))
    visited[r,c]=1
    
    effort = 0
    return find_path(heights_arr,visited,r,c,effort,max_row, max_col,min_effort)

#heights = [[1,2,2],[3,8,2],[5,3,5]]
heights = [[1,2,3],[3,8,4],[5,3,5]]
#heights = [[1,2,1,1,1],[1,2,1,2,1],[1,2,1,2,1],[1,2,1,2,1],[1,1,1,2,1]]
#heights = [[1,10,6,7,9,10,4,9]]
#heights = [[4,3,4,10,5,5,9,2],[10,8,2,10,9,7,5,6],[5,8,10,10,10,7,4,2],[5,1,3,1,1,3,1,9],[6,4,10,6,10,9,4,6]]
print(minimumEffortPath(heights))
#print(min_effort)

going from 0,0 to 1,0
effort before recurse: 0
going from 1,0 to 2,0
effort before recurse: 2
going from 2,0 to 2,1
effort before recurse: 2
going from 2,1 to 2,2
effort before recurse: 2
-------------------Found end: effort is 2
global min_effort after recurse: 2
going from 2,1 to 1,1
effort before recurse: 2
going from 1,1 to 1,2
effort before recurse: 5
going from 1,2 to 2,2
effort before recurse: 5
-------------------Found end: effort is 5
global min_effort after recurse: 2
going from 1,2 to 0,2
effort before recurse: 5
going from 0,2 to 0,1
effort before recurse: 5
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
going from 1,1 to 0,1
effort before recurse: 5
going from 0,1 to 0,2
effort before recurse: 6
going from 0,2 to 1,2
effort before recurse: 6
going from 1,2 to 2,2
effort before recurse: 6
-------------------Found end: effort is 6
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
going from 1,0 to 1,1
effort before recurse: 2
going from 1,1 to 2,1
effort before recurse: 5
going from 2,1 to 2,2
effort before recurse: 5
-------------------Found end: effort is 5
global min_effort after recurse: 2
going from 2,1 to 2,0
effort before recurse: 5
global min_effort after recurse: 2
global min_effort after recurse: 2
going from 1,1 to 1,2
effort before recurse: 5
going from 1,2 to 2,2
effort before recurse: 5
-------------------Found end: effort is 5
global min_effort after recurse: 2
going from 1,2 to 0,2
effort before recurse: 5
going from 0,2 to 0,1
effort before recurse: 5
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
going from 1,1 to 0,1
effort before recurse: 5
going from 0,1 to 0,2
effort before recurse: 6
going from 0,2 to 1,2
effort before recurse: 6
going from 1,2 to 2,2
effort before recurse: 6
-------------------Found end: effort is 6
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
going from 0,0 to 0,1
effort before recurse: 0
going from 0,1 to 1,1
effort before recurse: 1
going from 1,1 to 2,1
effort before recurse: 6
going from 2,1 to 2,2
effort before recurse: 6
-------------------Found end: effort is 6
global min_effort after recurse: 2
going from 2,1 to 2,0
effort before recurse: 6
going from 2,0 to 1,0
effort before recurse: 6
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
going from 1,1 to 1,2
effort before recurse: 6
going from 1,2 to 2,2
effort before recurse: 6
-------------------Found end: effort is 6
global min_effort after recurse: 2
going from 1,2 to 0,2
effort before recurse: 6
global min_effort after recurse: 2
global min_effort after recurse: 2
going from 1,1 to 1,0
effort before recurse: 6
going from 1,0 to 2,0
effort before recurse: 6
going from 2,0 to 2,1
effort before recurse: 6
going from 2,1 to 2,2
effort before recurse: 6
-------------------Found end: effort is 6
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
global min_effort after recurse: 2
going from 0,1 to 0,2
effort before recurse: 1
going from 0,2 to 1,2
effort before recurse: 1
going from 1,2 to 2,2
effort before recurse: 1
-------------------Found end: effort is 1
global min_effort after recurse: 1
going from 1,2 to 1,1
effort before recurse: 1
going from 1,1 to 2,1
effort before recurse: 4
going from 2,1 to 2,2
effort before recurse: 5
-------------------Found end: effort is 5
global min_effort after recurse: 1
going from 2,1 to 2,0
effort before recurse: 5
going from 2,0 to 1,0
effort before recurse: 5
global min_effort after recurse: 1
global min_effort after recurse: 1
global min_effort after recurse: 1
going from 1,1 to 1,0
effort before recurse: 4
going from 1,0 to 2,0
effort before recurse: 5
going from 2,0 to 2,1
effort before recurse: 5
going from 2,1 to 2,2
effort before recurse: 5
-------------------Found end: effort is 5
global min_effort after recurse: 1
global min_effort after recurse: 1
global min_effort after recurse: 1
global min_effort after recurse: 1
global min_effort after recurse: 1
global min_effort after recurse: 1
global min_effort after recurse: 1
global min_effort after recurse: 1
1

 

Notebook containing implementation with global vs no global variable: https://gist.github.com/gitgithan/a818d336c2309852a21d99efd619238d
My backtracking solution was to use a visited 2D array to track visits.Starting from the top-left cell, we try to move in the order of down, right, up, left (this order reaches the destination faster than some other orders, so the global min can be updated earlier, and used to bound/speed up the exploration of future paths) if within rectangle bounds and not visited before. Before moving (recursing), the visit is marked so future attempts to visit will return False for if valid().
Along the way, the effort (intialized to 0) of the path is updated using abs difference between the cell we are going to visit and current cell. If we reach base case (bottom-right cell), return the effort.
Once a single direction of recursion is complete, we update the global minimum min_effort (initialized to math.inf) and unmark the cell in that direction.
Once all 4 directions from the current cell is done, we return math.inf back up the call stack so it does not wrongly influence the min() of the caller.
This output of five 2, followed by two inf was mindboggling. It looked like the global variable min_effort which was updated from inf to 2 after the first solution (1, down to 3, down to 5, right to 3, right to 5) went back to inf! The final answer(path of 1, 2, 3, 4, 5) was also wrongly inf instead of 1.
Those five 2 correspond to the algorithm backtracking from destination 5 to 4,3,2,8,3,5 because all these have all 4 directions either visited or out of bounds, so the next path to try is to go right from 1,0 to 1,1 (3 to 8).
The problem is with the line min_effort = min(min_effort,find_path(...)) Because python evaluates arguments from left to right, the global variable min_effort was already evaluated before entering the recursion, so whatever happens in the recursion (including updates to global variable) has no effect on min_effort.
At the scope of the 3 in bottom middle of the grid, the algorithm has not yet found a successful path (the 1st path being 1,3,5,3,5), so min_effort was still math.inf at that moment, and min(math.inf,math.inf) became inf.
Swap arguments
Swap arguments
A fix is to simply swap the order of arguments to min_effort = min(find_path(...), min_effort). This gives the recursion a chance to update the global variable first before comparison.
2. Save into variable first
If you still wanted to maintain the old order as a matter of preference, the recursion can be calculated and saved into a variable first.
recursion_res = find_path(...)min_effort = min(min_effort, recursion_res)
3. Remove global variables completely
This was prompted by Leetcode giving wrong test results running all test cases, but running the failing case individually as custom input gives the correct result. If you insist on using globals on leetcode, copy the result, clear the global, and return the copied result so the cleared global is ready for the next test case.
Two changes: 1. Add the previously global variable to recursive function signature find_path(...,min_effort)
2. Return min_effort rather than math.inf
This will ensure the minimum effort path is always propagated through the recursive calls. If you ask why not return min_effort while using global variable, that is workable too, but unnecessary, since if you avoid the gotcha mentioned above, min_effort would have been updated to a non-inf value before the min(min_effort,find_path(...))comparison, so even if the 2nd argument returns math.inf, the 1st argument will always be updated to the correct value once any path first reaches the destination. Also, math.inf felt more representative of a “no solution/bad value”.
However, the argument in support of return min_effort rather than return math.inf is that it more accurately represents the most updated program state. Also, we can totally avoid the gotcha handling (the 1st two fixes described) and use min(min_effort,find_path(...)) directly because with return min_effort the 2nd argument will always be updated, so a “pre-recursion evaluated” min_effort that could contain its initialized math.inf in the 1st argument causes no harm. (exactly opposite situation of the previous paragraph).
Since we are looking for the global minimum, any path can be influenced by the results of any other path. This offers a chance for the current best result to short-circuit future paths once they are found to have longer effort than the current min_effort . This can be achieved by wrapping the backtracking section (visit,recurse,unvisit) with an extra check if updated_effort < min_effort: to largely reduce exploration steps, using “less than” because there is no point trying something that won’t reduce the current best.
if updated_effort < min_effort:                visited[row,col] = 1                min_effort = min(min_effort,find_path(heights_arr,visited,row,col,updated_effort,max_row,max_col))                visited[row,col] = 0
Global pros:
Lax return values allowed ( math.inf or min_effort )Convenient coding (less parameter passing and simpler recursive functions)
Lax return values allowed ( math.inf or min_effort )
Convenient coding (less parameter passing and simpler recursive functions)
Global cons:
Don’t gel with online coding platforms (risky for job interview automated coding assessments)Constantly have to be aware of side effects and variables not within current stackSensitive to function argument ordering (the root of all these troubles)
Don’t gel with online coding platforms (risky for job interview automated coding assessments)
Constantly have to be aware of side effects and variables not within current stack
Sensitive to function argument ordering (the root of all these troubles)
Local pros:
Easier debugging/tracing focusing on local problem spaceLess side effects
Easier debugging/tracing focusing on local problem space
Less side effects
Local cons:
More verbose function signature (more items to track means longer signature)Lose ability to return math.inf , have to return min_effort to propagate updated results back to rest of program
More verbose function signature (more items to track means longer signature)
Lose ability to return math.inf , have to return min_effort to propagate updated results back to rest of program
When using globals, if I had returned min_effort or coded properly using fix 2 above initially, I would not have seen this gotcha at all. It’s a blessing in disguise that returning the most intuitively correct value and being lazy with (possibly) extraneous variable assignments allowed me to strengthen my understanding of python evaluation order and debugging skills.