橫濱港
Goto was born in Kokufu-mura, Naka District, Kanagawa Prefecture. He was the eldest son of Izaemon Kobayakawa. He had three brothers and two sisters. After receiving an education, Goto worked as a city employee at the Port of Yokohama. It was there that he learned the English language.