Foreign Minister Stephen Smith told Australia 's national radio early Wednesday that Japan has not yet handed the men back to their own ship .