You just have to delete the photo from your travel diary. After that it's also removed from the photo book.