You probably understand that if your ex doesn't return with your child, you'll have a very difficult time getting him back as long as he remains in Nigeria. Then, if possible, both of you should sign the agreement before a notary public. At the very least, if something goes wrong, you'll have clear proof that the absence is not due to some misunderstanding. It may cost you a few bucks, but I'm sure your ex will be willing to pay if the alternative is for you to deny permission. Now I would like to regain custody of the children and can't afford a lawyer.
Every state has a system both for removing and restoring children from a parent's custody.
But the agency also has a duty to act in the best interests of the children. If staffers at the agency have become convinced that you don't have it in you to be a good parent, they may well expend very little effort trying to reunite you with your children. This leaves it up to you to force the agency to perform its job. If you can scrape together enough money for a consultation with a lawyer who specializes in this type of law, it would be well worth the scraping. In addition, you should sit down with an agency representative and make sure you understand what behavior on your part would make it more likely that you'll get your kids back.