Not only expensive, but it also has to be small enough to be physically close to the CPU (like on the chip).  Also, a lot of times the cache reads ahead, so it tries to predict what data will be accessed next, not just keeping what you've already requested.