At the time , he temporarily handed power to his younger brother , Defense Minister Raul Castro .