Q:

Too much memory getting used

I have been feeling this way for a while, looking at task manager mentally ballparking how much physical memory should be in use, and noticing a large disparity.  I finally got around to measuring it.  Taking the results of tasklist and adding, gave about 3.87 GB.  Task manager told me I was using 4.6 GB of memory.  This seems like a large disparity.  Where did it go? 
EDIT: So its clear, it is not going to prefetch.  It has also been suggested that its part of the OS, but not part of any of the system processes.  If this is true, please find a way to get the OS to tell me how much memory is getting used.  Since I want to find out where the memory went, asking me to look at the difference does nothing for me. 
EDIT 2: It is not the following:

The kernel
The cache
Hardware reserved memory

Picture of resources taken up.  Not quite sure what it will do but...

A:

Windows 7 uses 'unused' memory to precache things to make things run faster.  Should you need this memory, it'll get freed up.  
Its a feature, not a bug ;)
From technet

After you’ve used a Windows Vista system a while, you’ll see a low number for the Free Physical Memory counter on Task Manager’s Performance page.  That’s because SuperFetch and standard Windows caching make use of all available physical memory to cache disk data.  For example, when you first boot, if you immediately run Task Manager you should notice the Free Memory value decreasing as Cached Memory number rises.  Or, if you run a memory-hungry program and then exit it (any of the freeware “RAM optimizers” that allocate large amounts of memory and then release the memory will work), or just copy a very large file, the Free number will rise and the Physical Memory Usage graph will drop as the system reclaims the deallocated memory.  Over time, however, SuperFetch repopulates the cache with the data that was forced out of memory, so the Cached number will rise and the Free number will decline. 

A:

The numbers you see are totally imprecise and should only be taken as indications.  
An operating system is an extremely complex entity and a simple-minded program like Task Manager can never total the entire usage, many parts of it are simply inaccessible in User mode under which it runs.  Some of the elements that are inaccessible to it, or are simply impossible to attribute to any "task", are the hardware segmentation tables, shared-memory allocations, kernel-allocated memory, page-memory allocations, device-memory and many more. 
You will find more explanations in How do I interpret the Performance tab of Task Manager?,  but I choose to quote this part :

What do you expect from a program
  calling itself the "Task Manager"
  anyway?  There is nothing called a
  "task" in the operating system kernel
  - the OS has "processes" and "threads".  DOS had "tasks".  The Intel
  hardware has "task" structures, but
  the OS doesn't use them because it's
  faster for it to do it itself. 

The Task Manager is a very old Windows program, has problems even attributing functions to services, and the numbers it shows should be taken with a large pinch of salt.  It is probably missing out on large areas of memory that it cannot manage to count or attribute to any "task". 

A:

I suggest downloading the Sysinternals Suite from Microsoft.  It includes several utilities that will give you more information than Task Manager does.  Take a look at the Process Explorer, RAMMap, VMMap, and Process Monitor utilities.  To get the most information from them, you may need to do some configuration to change what info is displayed. 
Also, since you're interested in Windows memory, here is a series of blog posts from Mark Russinovich, one of the Sysinternals developer/founder.  It will help make sense of the info the utilities provide.