If you read primarily for entertainment, perhaps you could benefit from it. Another way I probably read differently from you is the fact that I read a bit more like an editor since I've begun writing. By that, I mean I don't necessarily finish reading everything I start. If a story or novel doesn't grab me in the first few pages, I usually speed up and skim ahead looking for something interesting to happen in the plot. If the story doesn't pick up, I drop it.
Life is too short to waste reading boring stuff.
I know there are certain writers who build a story slowly, and I must admit that I usually bail out on such stories. Some writers also put a lot of effort into crafting deeper meanings underneath the superficial plot, but that effort is often wasted on me. It's not them, it's me. I also read a great number of short stories, which is something you probably don't do. Perhaps a bit more than half of my fiction reading is short work found in magazines and anthologies.