Edo Castle
The character "niwa" in their name, meaning garden or yard, refers to the rumor that they were quartered in the garden of Edo Castle.