Fix the potential crash issue when switching map