Two were allowed to "escape" to the North in 1822, and two were granted freedom by his will upon his death in 1826