Remove offending FFI require

I have no idea how to detect that torch-ffi is installed, any advice welcome!