We have not really hurt them, but we have been foolish enough to hurt ourselves. This is a term we derive from John Stuart Mill. Admittedly, your motive is self directed to your own self. This can be an engrossing and low-level appetite for pleasure, which meaning the word hedonism has acquired. The argument cannot properly be against seeking happiness as an end, but only against certain kinds of happiness.
Hedonism generally has a bad name.
But happiness is still the end in view. Eudaemonism is merely an upgrading of hedonism. This does not limit action solely to self-regarding interests. In this case the choices may be designed for the happiness of others at the cost of the self. But the important thing to note is that the choices are by the person himself.