Takahashi was born to a samurai-class household at the Edo residence of Sano Domain, a subsidiary han of Sakura Domain, where his father was a retainer of the Hotta clan. Interested in art from childhood, he apprenticed to the Kanō school, but later became fascinated with western-style art through lithographs which were being available in Japan during the Bakumatsu period. In 1862, he obtained a place in the arts department of the "Bansho Shirabesho", the Tokugawa shogunate’s research institute in western learning, where he studied under Kawakami Togai, and where he began experimentation with oil painting. In 1866, he went to Yokohama to study under the English artist and cartoonist Charles Wirgman, who was so impressed with his talent that he sponsored his participation in the Paris World Exhibition of 1867.