Make sure the whisper dir gets the proper permissions as well