Your situation is not that uncommon.  Many women who had implants when they were younger decide years later that they want to go smaller or just get rid of their implants all together.
Removing implants is surprisingly straight forward and can be done under local anesthesia with no down time if they are not ruptured and if you have saline implants, it is a very simple procedure.
If you were extremely small to start with you may want something instead of going without.
In your case, at 5’9 and 120 pounds it is unlikely that you will have enough fat to get a reasonable result with fat grafting. In addition, it often takes two surgical sessions to achieve a noticeable or reasonable result.  You might consider a small natural feeling gummy bear implant as many women love these after having a saline implant.
Here is an article on a “mini boob job” that you might find helpful.