How fast does it need to be?  I don't think uploaded files are going to get massive amounts of traffic.  If they do at some point in the future, maybe we could add a 'make public' flag for the Media and have the view add proxy caching headers when returning those files.