Sabo most likely left Luffy and Ace as kids due to his influential parents coming after him, which he intended to stay away from.