Generally, if someone is prohibited from owning firearms that extends to possessing them or having access to them. This means that none should be on the premises in which they live, no mater who the legal/registered owner is. Accordingly, you should remove them your guns until such time as your brother moves out.