You can only remove the Fia debuff by finishing your song with 99% or higher accuracy.