The colony of New Amsterdam was centered at the site which would eventually become Lower Manhattan