This is because of the way git's submodule handing works; I could change it so that I always cd, but not the other way around.