Worse, that bias distorts the significance that a work like HoD has or can have for you. Every economic advantage you enjoyed, even, and particularly, the modest middle class, was due to the imperialist control of your Empire. The problem with that worldview, as I see it, is that is that it relies on classifying people into groups rather than on seeing them as individual humans. For this, Europeans and Americans share immeasurable responsibility.
Imperialism exists and existed.
Our feelings about it, guilty or not, are quite immaterial. Our individual feelings do indeed matter, because they are what allow us to grow and to change. They make us human. The function of the humanities is to assert and define our humanity, and the sort of analysis you offer is antithetical to that. I for one reject it, and I strongly suspect that Conrad would share my view.