Wu Garden was built around 1828 when salt magnate Wu Shangxin bought the garden of He Bin (), an interpreter during the era of Dutch rule who played a role in the fall of Fort Zeelandia. At the time, it was also called "lâu-á lāi (樓仔內)". Alluding to the wealth of the Wu family, there was a local saying "Even if you have the wealth of "lâu-á lāi", you don't own the estate of "lâu-á lāi" ; even if you own the estate of "lâu-á lāi", you don't have the wealth of "lâu-á lāi" 「有樓仔內的富，也無樓仔內的厝；有樓仔內的厝，也無樓仔內的富」".