Although many shops and department stores can provide you with individual components, it's best to establish a relationship with a single business that can supply everything you need. Make sure the business you choose can provide not only the basic hardware, but also the various hops, malts, yeasts, adjuncts, herbs, publications, and odd bits of stuff you'll need to homebrew successfully. If you live in a more rural locale, you may have to acquire your supplies through mail-order. Unless you already know a bit about homebrewing, your first purchase should be a handbook or manual covering the basics. One standby is Charlie Papazian's Homebrewer's Companion.
Plain bleach is best.
A large pot that can hold a least six quarts of liquid. Stainless steel is recommended. This pot shouldn't be used for anything else. These bottles are difficult to cap, and they break easily. You'll need at least two cases of bottles for every batch of beer, possibly more.