Depends on what you mean by refuse.  You can be subpoeaned and 'compelled' to testify but no one but you controls what comes out of your mouth.  If you are legitimately fearful let the appropriate people know and they may be able to accomodate you.