The hidden cost of performance tweaks is that your code becomes less understandable and maintainable. Remember that code is read much more than it is written. They will help both you and everyone who comes after you. If nothing else, the frustration of trying to ferret out useful information from the online Java documentation should convince you. When you think you've got a good analysis, design, or implementation, do a walkthrough.
Elegance always pays off.
And there's nothing that matches the feeling that comes from knowing you've got an amazing design. You can find other programming guidelines on the Web.