Yes, it is absolutely safe to use Google Drive with the Files app on iOS. You have access to your Drive files as long as you have the Drive app installed on your iPhone with your Google account logged on it. As soon as you delete the Drive app or log out from your Google account, the access to your Drive folder will be revoked from the Files app.