The short answer to this question is "It depends". While we're not used to eating any other part of a fish except for its flesh, there are plenty of people who consume fish eyeballs, for instance. So the answer is - leave any parts of the fish that you consider edible.