Things just weren't working out for him on Wall Street