Move cffi load into __init__ functions

Like mentioned on slack saw no issues created by this change.