You can refuse a back to work interview, but it will do you no good. You should not be nervous about the interview, as your manager’s purpose is to understand the reasons behind your absence and how he/she can help you transition better and faster.