It is not the fruit of the tree that conveyed death. The fruit of that tree was not ontologically different from the fruit of any other tree. The power of the tree was in the covenant. Their eating was a judicial act that included certain covenantal consequences because of the word of God that instituted that covenantal arrangement. Consider the analogy of taking human life.
Murder is a certain sort of taking of human life.
It is defined by law, not by the physical act itself. If I take the life of an enemy in war, I am praised. It is the law that defines the act of murder. When one commits the act defined by law as murder, even though the actual physical deed may be the same as what one does in a time of war, the meaning of the deed is totally different. The only act Adam performed was to eat.